My Notes

Table of Contents

1. PHP

1.1. Installation

1.1.1. Debian

Install multiple versions of PHP 

# install an apt repo which allows you to switch PHP versions
# installed on Ubuntu 18.04
sudo apt install software-properties-common
sudo add-apt-repository ppa:ondrej/php

sudo apt-get update

# install PHP for Apache (install Apache and everything?)
sudo apt install php7.3

# install PHP as FPM (one type of FastCGI, full name FastCGI Process Manager). Needed for Nginx
sudo apt install php7.3-fpm

# install a specific module, use tab
sudo apt install php7.3-cli

# install multiple modules
# sudo apt install php7.3-cli php7.3-xml php7.3-mysql
sudo apt install php7.3-{cli,xml,mysql,zip,mbstring,curl}

# after install, php-fpm services should be auto started
sudo systemctl status php7.3-fpm

# change PHP version as an Apache module only
a2dismod php7.0 ; a2enmod php5.6 ; service apache2 restart

# check PHP version
php -v

# see what alternatives of PHP
update-alternatives --config php

# set default PHP version for PHP CLI only
sudo update-alternatives --set php /usr/bin/php7.3

# get location of php.ini, usually `/etc/php/7.3/cli/php.ini`
php -i | grep "Loaded Configuration File"

1.1.2. Xdebug

apt update
apt install php7.4-xdebug

# find which xdebug ini file is loaded
php --ini | grep xdebug

# e.g. /etc/php/7.4/cli/conf.d/20-xdebug.ini

# /etc/php/7.4/cli/conf.d/20-xdebug.ini
# and
# /etc/php/7.4/fpm/conf.d/20-xdebug.ini both are symlink to:
# /etc/php/7.4/mods-available/xdebug.ini

# add the following line to /etc/php/7.4/mods-available/xdebug.ini
1.1.2.1. Settings
zend_extension=xdebug.so
xdebug.mode=debug,profile

# xdebug.start_with_request=default means trigger for debug, yes for profile
xdebug.start_with_request=trigger

# php -d xdebug.profiler_enable=On script.php

xdebug.idekey=docker

# The internal IP of the docker host (Mac). Eq. `ipconfig getifaddr en0`
xdebug.client_host=host.docker.internal

# https://xdebug.org/docs/profiler

# default. %p for PID. Other %s for script location
# xdebug.profiler_output_name=cachegrind.out.%p

# https://xdebug.org/docs/profiler ends

# Xdebug v3 uses port 9003 as default
# xdebug.client_port=9003

# # Xdebug introduces nesting level e.g. recursion. Default is 256
# xdebug.max_nesting_level=999999

# create a webpage that runs:
# <?php
# var_dump(xdebug_info());

# Or
# php --ri xdebug

# Xdebug version: php -v
1.1.2.2. Profiling
Output directory
php --ri xdebug | grep xdebug.output_dir
default
/tmp
Install an app on MacOS
brew install qcachegrind
Then open a profiling file
qcachegrind cachegrind.out.1
(no term)
Metrics
  • 100,000 * 10ns = 1ms
  • Lower right panel
    Callees
    callees at distnace 1 (first level)
    Call graph
    significant callers and callees at different distance
    All Callees
    callees at all distance
1.1.2.3. PhpStorm set up
  • Xdebug is installed inside a docker with PHP with port 80 exposed and mapped to docker host port 80
  • https://dev.to/jackmiras/xdebug-in-phpstorm-with-docker-2al8
  • Make sure Settings > Build, Execution, Deployment > Docker is connection successful
    • Docker for Mac: default unix:///var/run/docker.sock
    • This Docker setting is shared across projects in PhpStorm
  • Create a Server on PhpStorm (A debugging server on local)
    • Settings > PHP > Servers
      • Host: localhost
      • Port: 80
      • Debugger: Xdebug
      • Use path mappings
  • Add PHP Remote Debug
    • On the main debug tool, click on 3 vertical dots and choose edit
    • Add New Configuration > PHP Remote Debug
      • Filter debug connection by IDE key: check
        • Server: the server we just created above
        • IDE key (session id): docker
  • Click Debug button
  • Extra settings
    • PHP > Debug
      • PHP > Debug > Xdebug
        • Force break at first line when no path mapping specified: uncheck
        • Force break at first line when a script is outside the project: uncheck
      • PHP > Debug > Settings
        • Notify if debug session was finished without being paused: uncheck

1.2. Configuration & CLI

1.2.1. CLI

Get php configuration files location
php --ini
  • <?php var_dump(php_ini_loaded_file()); ?>
(no term)
Get loaded extensions/modules
  • php -r "print_r(get_loaded_extensions());"
    -r
    run PHP code without using script tags <?..?>
  • show configuration for extension xdebug
  • all loaded modules
    php -m | grep -e redis
    see if a module is installed
Get a php config variable
php -i | grep --color -n "memory_limit"
  • <?php echo ini_get('memory_limit'); ?>
All CLI options
https://www.php.net/manual/en/features.commandline.options.php
php -i
eq. to phpinfo()
php -d xdebug.profiler_enable=On script.php
override INI settings
Interactive shell mode
php -a, type quit
1.2.1.1. Command line arguments
php script.php value1 value2

if (isset($argc)) {
    for ($i = 0; $i < $argc; $i++) {
        echo "Argument #" . $i . " - " . $argv[$i] . "\n";
    }
}
else {
    echo "argc and argv disabled\n";
}

/*
Argument #0 - script.php
Argument #1 - value1
Argument #2 - value2
*/

# assign command line arguments as $_GET
php -f a.php a=1 b[]=2 b[]=3

// a.php
parse_str(implode('&', array_slice($argv, 1)), $_GET);

$_GET === [
    'a' => '1',
    'b' => ['2', '3'],
];

// URL: ?id[gte]=2&id[]=1&id[]=3
$_GET === [
    'id' => [
        'gte' => 2,
        0 => 1,
        1 => 3
    ],
];

// Wait for input
echo "Are you sure you want to do this?  Type 'yes' to continue: ";
$handle = fopen ("php://stdin","r");
$line = fgets($handle);
if (trim($line) != 'yes') {
    echo "ABORTING!\n";
    exit;
}
fclose($handle);
echo "\n"; 
echo "Thank you, continuing...\n";

$fieldA = (function ($value) {
    $result = ($_f = function ($fieldA = null) use (&$_f) {
        if (! intval($fieldA)) {
            echo "Please provide a fieldA (integer):\n";
            $handle = fopen('php://stdin', 'r');
            $line = intval(trim(fgets($handle)));
            fclose($handle);
            $fieldA = (! $line) ? $_f() : $line;
        }
        return intval($fieldA);
    })($value);
    echo "fieldA is set to {$result}\n";
    return $result;
})($argv[1] ?? null);

abc=1 php a.php

$abc = getenv('abc');
$abc = ($abc === false) ? 'not exists' : $abc;

1.2.2. Basics

  • Changing php.ini requires PHP restart and apache:restart or Nginx 

    # raw way
/etc/init.d/php7.4-fpm restart

# SystemD
systemctl | grep 'php'
systemctl restart php7.4-fpm.service
1.2.2.1. PHP Version
phpinfo();
echo phpversion();
echo PHP_VERSION;
1.2.2.2. Possible places for php.ini
  • 7.x
    • Ubuntu
      Apache
      /etc/php/7.x/apache2/php.ini, /etc/php/7.0/apache2/conf.d/*.ini
      CLI
      /etc/php/7.x/cli/php.ini, /etc/php/7.x/cli/conf.d/*.ini
    • Debian
      Nginx (FPM)
      image:php:7.x-fpm:ini
  • /usr/local/etc/php/php.ini, /usr/local/etc/php/conf.d/*.ini
  • /etc/php5/apache/php.ini, /etc/php5/cli/php.ini
1.2.2.3. php.ini vs .user.ini

Put .user.ini in website root directory. Only place php.ini in main config or conf.d directory or a directory that is setup to scan php.ini files. Placing php.ini in website root directory only works for that directory but not recursively in sub directories.

1.2.2.4. Apache
  • Set PHP via Apache directives e.g. httpd.conf .htaccess
  • Require AllowOverride Options or AllowOverride All
  • Cannot use PHP constants e.g. E_ALL
  • Changing this does not require to reload Apache nor PHP

And then 

// for value. Can be used only with PHP_INI_ALL and PHP_INI_PERDIR type directives
php_value name value

// for boolean. Can be used only with PHP_INI_ALL and PHP_INI_PERDIR type directives
php_flag name on|off

// This cannot be set in .htaccess file and cannot be overriden by .htaccess or ini_set()
php_admin_value name value

// This cannot be set in .htaccess file and cannot be overriden by .htaccess or ini_set()
php_admin_flag name on|off
1.2.2.5. Configuration Mode
PHP_INI_USER
Entry can be set in user scripts (like with ini_set()) or in the Windows registry. Since PHP 5.3, entry can be set in .user.ini
PHP_INI_PERDIR
Entry can be set in php.ini, .htaccess, httpd.conf or .user.ini (since PHP 5.3)
PHP_INI_SYSTEM
Entry can be set in php.ini or httpd.conf
PHP_INI_ALL
Entry can be set anywhere
1.2.2.6. Directive Value

Value can be

  • a string

    • Empty string 

      foo =         
; sets foo to an empty string
foo = None
; sets foo to an empty string
foo = "None"
; sets foo to the string 'None'
  • a number
  • e.g. E_ALL or M_PI
  • On, Off, True, False, Yes, No and None
  • an expression
    • e.g. E_ALL & ~E_NOTICE
    • a quoted string ("bar")
    • e.g. ${foo}
    • Expressions in the INI file are limited to bitwise operators and parentheses:

      • | bitwise OR 

        define('A',1);
define('B',1<<1);
define('C',1<<2);

$a = A;
$b = $a | B;
$c = $b | C;

$c2 = A;
$c2 |= B;
$c2 |= C;

$d = A | B | C;

$c === $d;
$c === $c2;
      • ^ bitwise XOR
      • & bitwise AND
      • ~ bitwise NOT
      • ! boolean NOT
  • Boolean
    True
    use 1, On, True or Yes
    False
    use 0, Off, False or No

1.2.3. [PHP]

http://php.net/manual/en/ini.list.php

1.2.3.1. magic quotes

Ignore. They are deprecated. Should be Off at all times 

; Magic quotes for incoming GET/POST/Cookie data.
magic_quotes_gpc = Off

; Magic quotes for runtime-generated data, e.g. data from SQL, from exec(), etc.
magic_quotes_runtime = Off

; Use Sybase-style magic quotes (escape ' with '' instead of \').
magic_quotes_sybase = Off
1.2.3.2. allow_url_fopen

Default is 1 to enable. If disabled, file_get_contents is not possible. This setting can only be set PHP_INI_SYSTEM

1.2.3.3. auto_prepend_file
// default
auto_prepend_file =

auto_prepend_file = /var/www/prepend.php
if (isset($_SERVER['HTTP_X_FORWARDED_FOR'])) {
    $_SERVER['REMOTE_ADDR'] = $_SERVER['HTTP_X_FORWARDED_FOR'];
}
1.2.3.4. expose_php

expose_php = On

1.2.3.5. session
1.2.3.6. extension_dir php.ini:extension_dir
1.2.3.7. max_execution_time php.ini:max_execution_time
  • PHP_INI_ALL, default 30s
  • set_time_limit( int $seconds ) can extend the limit by $seconds for one PHP script file
1.2.3.8. include_path
  • See 1.9.3
  • use : to separate include_path = ".:/usr/share/php"
  • use ; to separate include_path = ".;c:\php\includes"
  • get_include_path()
  • set_include_path()
1.2.3.9. short_open_tag

short_open_tag = Off

1.2.3.10. memory_limit default 128M PHP_INI_ALL 
// in bytes. Corresponds to php.ini:memory_limit
var_dump(memory_get_peak_usage());

1.2.4. Enable extension php.ini:extension_dir

php.ini can also enable module. e.g. to enable module intl. Use php.ini:extension_dir 

;   extension=modulename
; For example:

extension=mysqli
extension=/path/to/extension/mysqli.so

; xdebug
zend_extension = /path/to/extension/xdebug.so

http://php.net/manual/en/extensions.alphabetical.php https://pecl.php.net/packages.php

1.2.5. prod 

; Show all errors, except for notices and coding standards warnings.
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
display_errors=Off
log_errors=On

; error_log not set go to Apache error log
; Refer to apache:core:serversignature 
expose_php = Off

memory_limit = 128M
post_max_size = 8M
max_execution_time = 30
max_input_time = 60

; default no functions are disabled
; disable_functions = show_source, exec, shell_exec, system, passthru, proc_open, popen

file_uploads = On
max_file_uploads = 20
upload_max_filesize = 2M

allow_url_fopen = On

; Whether to allow `include`/`require` to open URLs (like http:// or ftp://) as files.
allow_url_include = Off

; php:session

; in seconds
session.cookie_lifetime = 0
session.cookie_secure = 1 ; 0 if not using SSL
session.cookie_httponly = 1
session.use_only_cookies = 1 ; ID can't come from GET or POST
; session.entropy_file = "/dev/urandom" ; add more randomness bits

session.gc_maxlifetime=1440 ; seconds, default. Unused PHP session will be kept alive for 24 miutes.

1.2.6. dev

1.2.7. Mail

http://php.net/manual/en/mail.configuration.php 

[mail function]
; For Win32 only.
; http://php.net/smtp
SMTP = localhost
; http://php.net/smtp-port
smtp_port = 25

; For Win32 only.
; http://php.net/sendmail-from
;sendmail_from = me@example.com

; For Unix only.  You may supply arguments as well (default: "sendmail -t -i").
; http://php.net/sendmail-path
;sendmail_path =

; Force the addition of the specified parameters to be passed as extra parameters
; to the sendmail binary. These parameters will always replace the value of
; the 5th parameter to mail().
;mail.force_extra_parameters =

; Add X-PHP-Originating-Script: that will include uid of the script followed by the filename
mail.add_x_header = On

; The path to a log file that will log all mail() calls. Log entries include
; the full path of the script, line number, To address and headers.
;mail.log =
; e.g. change to 
; mail.log = /var/log/phpmaillog

; Log mail to syslog (Event Log on Windows).
;mail.log = syslog

Enable mail.log, make sure /var/log/phpmaillog has www-data:www-data as the apache

You may need to change the sendmail_path to a php wrapper instead of just adding mail.log location http://www.matteomattei.com/how-to-log-email-sent-from-php-through-mail-function/ https://www.howtoforge.com/how-to-log-emails-sent-with-phps-mail-function-to-detect-form-spam

1.2.8. Error log

  • d7:show error
  • 2.5
  • wp:show error
  • Runtime configuration for error handling

  • Sample 

    ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL);
  • display_errors, display_startup_errors
    • PHP_INI_ALL. Default 1
    • PROD always uses Off or 0
  • log_errors
    • PHP_INI_ALL. default 0. prod needs to log error instead of display_errors. If true, either to server's system log or error_log
  • log_errors_max_len
    • PHP_INI_ALL. default 1024 (bytes). 0 to no limit
  • error_log
    • PHP_INI_ALL. default NULL or empty. place to store error log. If not set, errors are sent to SAPI error logger e.g. Apache error log or stderr in CLI
    • use whatever Linux syslog is using
  • error_reporting
    • PHP_INI_ALL
    • https://www.php.net/manual/en/errorfunc.constants.php
    • show all except deprecated and notice
      • eq. to E_ALL ^ (E_DEPRECATED | E_NOTICE)
    • eq. E_ALL & ~E_DEPRECATED

    • Temporarily set 

      $oldErrorReporting = error_reporting();
error_reporting( $oldErrorReporting & ~E_WARNING & ~E_NOTICE );
// ... some code that threw warning and notice before, now won't
error_reporting( $oldErrorReporting );
    • Refer to php:@
      • Whenever an error of any level is generated, it is formatted and ready to output to browsers. Only just before it, error_reporting setting is checked. This applies to php:@ as well
    • https://stackoverflow.com/a/1869185/2196360
      10k errors of notice level, with error_reporting and display_errors enabled
      5k ms
      same, but with display_errors disabled
      136 ms
      same, but with error_reporting disabled too
      118 ms
      no error of notice level
      20 ms

    • Example 

      // Rule of thumb, always set a valid value with the right type for a key/value if you want to manipute later
// Save you a lot of work to check, validate and type conversion hassles down the road

$x = [];
$a = $x['abc'] + 0; // notice undefined index

$x['abc'] = '';
$a = $x['abc'] + 0;
// fatal error for array
// warning non-numeric for string
// no warning for null

$a = intval($x['abc'] ?? 0); // PHP 7 using null coalescing operator. good!
$a = floatval($x['abc'] ?? 0); // PHP 7. good!
$a = (array)($x['abc'] ?? null);

$a = intval(isset($x['abc']) ? $x['abc'] : 0); // PHP < 7. good!
$a = floatval(isset($x['abc']) ? $x['abc'] : 0); // PHP < 7. good!

// Use this function to have a warning/error proofing and the result could be int or float!
// Use this also for PHP 7+ when the result type cannot not be predefined
function toNumber($valueOrKey, $array = null, $default = 0) {
    $val = $valueOrKey;
    if (is_array($array) && isset($array[$val])) {
        $val = $array[$val];
    }
    if (is_numeric($val)) {
        return $val + $default;
    }

    return $default;
}

var_dump($test = toNumber(null), $test === 0);
var_dump($test = toNumber(''), $test === 0);
var_dump($test = toNumber(array()), $test === 0);
var_dump($test = toNumber(array(1, 2)), $test === 0);
$x = array('a' => 123);
var_dump($test = toNumber('keyNotExists', $x), $test === 0);
var_dump($test = toNumber('a', $x), $test === 123);

function arrayValue($key, $array, $default = null) {
    return isset($array[$key]) ? $array[$key] : $default;
}

$a = arrayValue('abc', $x);
$a = isset($x['abc']) ? $x['abc'] : null;
// $a = $x['abc'];

if (isset($x['abc']) ? $x['abc'] : null) {}
if (arrayValue('abc', $x)) {}
// if (array_key_exists('abc', $x) && $x['abc']) {}
//if ($x['abc']) {}

if (is_numeric(arrayValue('abc', $x))) {}
// if (isset($x['abc']) && is_numeric($x['abc'])) {}
// if (is_numeric($x['abc'])) {}

1.2.9. Trigger error

  • Syntax 

    trigger_error ( string $error_msg [, int $error_type = E_USER_NOTICE ] ) : bool
  • set_error_handler

1.2.10. Custom error/warning/notice hanlder

  • https://www.php.net/manual/en/function.set-error-handler.php 

    // set_error_handler returns the previous handler
// Get error handler
set_error_handler($handler = set_error_handler('var_dump'));
var_dump('error handler', $handler); // NULL | string | array(2) | Closure

restore_error_handler(); // no parameter

// Set a handler for multipl error levels
$oldHandler = set_error_handler(
    function ($errno, $errstr, $errfile, $errline, $errcontext) {
        //var_dump('$errno', $errno); // int, same as error level
        //var_dump('$errstr', $errstr); // e.g. `Undefined index: someindexname`
        //var_dump('$errfile', $errfile); // absolute path to file
        //var_dump('$errline', $errline); // int
        //var_dump('$errcontext', $errcontext); // deprecated as of PHP 7.2.0 and removed in PHP 8
        $ignoreFiles = array('partial/path/to/file1', 'partial/path/to/file2');
        $r = false;
        foreach ($ignoreFiles as $ignoreFile) {
            if (strlen($ignoreFile) > 0 ? substr($errfile, -strlen($ignoreFile)) === $ignoreFile : true) {
                // file path ends with
                $r = true;
                break;
            }
        }

        return $r;
        // false: nothing will be output in this function and the default PHP error handler will be run
        // true: output as defined in this function and the default php error handler will not be run
    },
    E_NOTICE | E_WARNING
);

1.2.11. register_shutdown_function

  • Run after script execution finishes or exit() is called
  • Shutdown functions can run beyond max_execution_time

  • Example 

    register_shutdown_function('shutDownLogError');

function shutDownLogError() {
    $e = error_get_last();
    if (isset($e['type']) && $e['type'] & (E_ERROR | E_PARSE | E_CORE_ERROR | E_COMPILE_ERROR | E_USER_ERROR)) {
        $errstr = $e['file'] . '(' . $e['line'] . '): ' . $e['message'];
        badnews_logtodb($errstr);
    }
}

1.2.12. opcache

  • Preload Laravel
    • https://stitcher.io/blog/preloading-in-php-74

    • opcache.preload=/path/to/project/preload.php preload.php 

      $files = /* All files in eg. vendor/laravel */;

foreach ($files as $file) {
    // will parse but not execute
    opcache_compile_file($file);

    // to ensure linked files are also loaded, use require_once
    // require_once($file);
}
    • Load all dependecies e.g. parent class first
    • To verify, restart php-fpm and var_dump(opcache_get_status())

1.3. Kill

ps x
# find the pid
kill 1234

1.4. Constant define vs const

Predefined constants / reserved constants
https://www.php.net/manual/en/reserved.constants.php
(no term)
Magic constants
  • https://www.php.net/manual/en/language.constants.magic.php
  • string
  • current file directory path without trailing slash unless it's the root directory
  • current function name or {closure} for anonymous function
    • In case of closure
  • get_class() . ' ' . __LINE__ or
    • __TRAIT__ . ' ' . __LINE__
  • the class method name
  • __TRAIT__
Predefined variables
https://www.php.net/manual/en/language.variables.predefined.php
Reserved variables
https://www.php.net/manual/en/reserved.variables.php
Superglobals
https://www.php.net/manual/en/language.variables.superglobals.php
(no term)
Naming
  • No hyphen!
  • a-z, A-Z, and the ASCII characters from 128 through 255 (0x80-0xff)
(no term)
const
  • defines constant at compile time which means:
    • cannot const FOO = 'BAR'; inside a condition block
    • cannot define a constant using a global variable global $A; const ABC = $A;
      • Previous define and const can be used in the value
    • PHP 5.6+ array can be assigned and expressions are allowed
  • Can be used inside and outside of a class
  • Can define a constant using another constant

  • const defines a constant in the current namespace 

    namespace A\B\C; // To define the constant
const FOO = 'BAR';
// eq.
define('A\B\C\FOO', 'BAR');
(no term)
define
  • Defines constant at run time
  • can specify whether it's case sensitive or not (although it's deprecated in PHP 7.3.0)
  • Drawbacks
    • Scalar only for PHP 5.x, no array. 5 can define constants as array for PHP 5.6+

    • PHP 7.0+ can define array 

      define('ABC', array('a' => 1, 'b' => 2));
var_dump(ABC['a']);
    • Cannot be used inside a class
// define $GLOBALS['MIN_VALUE']
define('MIN_VALUE', '0.0');   // OUTSIDE of a class definition
define('MAX_VALUE', '1.0');   // OUTSIDE of a class definition

if (defined('MYGLOBALVAR') && MYGLOBALVAR !== 'abc') {}

const MIN_VALUE = 0.0;
const MAX_VALUE = 1.0;

1.5. Protocols

1.5.1. php://

php://stdin
read only
php://stdout
write-only
$fptr = fopen(getenv("OUTPUT_PATH"), "w");

$stdin = fopen("php://stdin", "rb"); // b means binary and has to be put at the end, t means plaintext

fscanf($stdin, "%d\n", $ar_count); // total number of different numbers

fscanf($stdin, "%[^\n]", $ar_temp); // numbers separated by space

$ar = array_map('intval', preg_split('/ /', $ar_temp, -1, PREG_SPLIT_NO_EMPTY));

$result = simpleArraySum($ar);

fwrite($fptr, $result . "\n");

fclose($stdin);
fclose($fptr);

1.6. $_SERVER

  • Which are safe?
    • GATEWAY_INTERFACE
    • SERVER_ADDR
    • SERVER_SOFTWARE
    • DOCUMENT_ROOT
    • SERVER_ADMIN
    • SERVER_SIGNATURE
  • Refer to 1.37

1.6.1. Partyly safe

HTTPS
bool whether the request is https
HTTP_HOST
see below
(no term)
REQUEST_TIME
REQUEST_URI
/abc/xyz.html?abc=xyz without hash. Raw value is encoded. Use urldecode
REQUEST_METHOD

$method = $_SERVER['REQUEST_METHOD'];
if ('PUT' === $method) {        
    parse_str(file_get_contents('php://input'), $_PUT);
    var_dump($_PUT); //$_PUT contains put fields 
}
(no term)
REMOTE_ADDR
REMOTE_HOST
  • relies on reverse DNS lookups and may hence be spoofed by DNS attacks against server (at that time you have bigger problems anyway)
  • Its value may be a proxy, which is a simple reality of the TCP/IP protocol and nothing you can do.
(no term)
REMOTE_PORT
(no term)
SERVER_PROTOCOL
SERVER_NAME
see below
SERVER_PORT
see below
SCRIPT_NAME
/index.php. Similar to $_SERVER['PHP_SELF'] which is only for PHP
SCRIPT_FILENAME
/index.php. Maybe a relative path when the script is executed with CLI
REMOTE_*
values are guaranteed to be the valid address of the client as verified by a TCP/IP handshake. Because it's the address the response will be sent to

1.6.2. HTTP_HOST and SERVER_NAME

  • curl -H "Host: notyourdomain.com" http://yoursite.com/
    • By default, HTTP_HOST or SERVER_NAME is notyourdomain.com
    • Server might hard set (static) SERVER_NAME to a real machine name e.g. Pantheon
    • Server might dynamically set HTTP_HOST based on the request e.g. Pantheon
    • Attackers might trigger a lot of requests and cause your website that is cached by a proxy (Varnish, Cloudflare) to have wrong cache that has content which points to the attacker's domain. Cache poisoning
    • Never display $_SERVER['HTTP_HOST'] or $_SERVER['SERVER_NAME']
    • For Pantheon, set $_SERVER['SERVER_NAME'] = $_SERVER['HTTP_HOST'];
$config['site_url'] = 'http://' . $_SERVER['HTTP_HOST'] . '/index.php';
$config['cp_url']   = 'http://' . $_SERVER['HTTP_HOST'] . '/system/index.php';

$domain = 'example.com';
$config['site_url'] = 'http://' . $domain . '/index.php';
$config['cp_url']   = 'http://' . $domain . '/system/index.php';

$domains = array('domain.com', 'dev.domain.com', 'staging.domain.com', 'localhost');
if (in_array($_SERVER['HTTP_HOST'], $domains)) {
    $domain = $_SERVER['HTTP_HOST'];
}
else {
    $domain = 'localhost';
}

1.6.3. SERVER_PORT

if (isset($_ENV['PANTHEON_ENVIRONMENT'])) {
  if (isset($_SERVER['HTTP_USER_AGENT_HTTPS']) && $_SERVER['HTTP_USER_AGENT_HTTPS'] === 'ON') {
    $_SERVER['SERVER_PORT'] = 443;
  }
  else {
    $_SERVER['SERVER_PORT'] = 80;
  }
}

1.7. $_FILES POST method uploads

  • http://www.php.net/manual/en/features.file-upload.post-method.php
  • <input type="file" name="userfile" >
    • If no name attribute, then it's file, which means $_FILES['file']
  • $_FILES['userfile']['name']
    • The original name of the file on the client machine
    • File extension e.g. jpg
      • $extension = strtolower(pathinfo($_FILES['userfile']['name'], PATHINFO_EXTENSION));
  • $_FILES['userfile']['type']
    • The mime type of the file, if the browser provided this information. An example would be "image/gif". This mime type is however not checked on the PHP side and therefore don't take its value for granted
  • $_FILES['userfile']['size']
    • size in bytes, of the uploaded file
  • $_FILES['userfile']['tmp_name']
    • Temporary filename of the file in which the uploaded file was stored on the server
  • $_FILES['userfile']['error']
    • Error code associated with this file upload. No error is 0

  • POST as array 

    <form action="" method="post" enctype="multipart/form-data">
<div>Pictures:
    <input type="file" name="pictures[]" />
    <input type="file" name="pictures[]" />
    <input type="file" name="pictures[]" />
    <input type="submit" value="Send" />
</div>
</form>

<?php
foreach ($_FILES["pictures"]["error"] as $key => $error) {
if ($error == UPLOAD_ERR_OK) {
    $tmp_name = $_FILES["pictures"]["tmp_name"][$key];
    // basename() may prevent filesystem traversal attacks;
    // further validation/sanitation of the filename may be appropriate
    $name = basename($_FILES["pictures"]["name"][$key]);
    move_uploaded_file($tmp_name, "data/$name");
}
}

1.8. $_COOKIE values may also exist in $_REQUEST

1.9. Filesystem

http://php.net/manual/en/ref.filesystem.php

1.9.1. Delete file

Use absolute path. See dirname 

$files = [
    './first.jpg',
    './second.jpg',
    './third.jpg'
];

foreach ($files as $file) {
    if (file_exists($file)) {
    unlink($file);
    } else {
    // File not found.
    }
}

1.9.2. File Modification time

filemtime(__DIR__. "/wp-content/themes/a.css")) // int

1.9.3. Include file __DIR__, dirname(__FILE__)

  • Change current working directory 

    echo getcwd() . "\n"; // /var/www/mysite/test
chdir(__DIR__ . '/../app/');
echo getcwd() . "\n"; // /var/www/mysite/app

  • An absolute path is defined 

    include __DIR__ . "/../abc.";

// PHP version < 5
include dirname(__FILE__) . "/../../../../../wp-content/themes/a-theme/includes/thank-you.php"; 

foreach ( glob( dirname( __FILE__ ) . '/abc/*.php' ) as $file ) {
    include $file;
}

  • start with ./ or ../ 

    cd /path/to/project # this is the entrypoint which is the result of getcwd(), current working directory
php run/start.php # this is not the entrypoint! the start.php file path

    include './connect.php';
// getcwd() . 'connect.php', which is /path/to/project/connect.php
// if not found, return error
// same applies to `include '../connect.php'`

  • a.php or inc/a.php 

    cd /path/to/project # this is the entrypoint
php run/start.php

    /path/to/project/run/start.php 

    include 'connect.php';

    • Try these in order

      default include paths have 2 directories
      . and /usr/share/php
      .
      stands for the entrypoint or current working directory getcwd(), which is /path/to/project so the file included is /path/to/project/connect.php. Which is the same behavior as relative path
      /usr/share/php
      /usr/share/php/a.php
      • If the file isn't found in 1.2.3.8
      (no term)
      the calling script's own directory
      (no term)
      the current working directory which is the same as . in 1.2.3.8
  • Entrypoint
    Command line
    the cd path which is /path/to/project in this case
    Apache/Nginx

    always the path to the nearest directory

  • cd into the directory which has the script file and run it
  • Summary
    • If the file to be included is in the same or deeper directory of the calling script's own directory, use bare file name or absolute path
    • If the file to be included is in the parent directory of the calling script's own directory
      • Try to avoid this happening
      • Use absolute path
      • When relative path is used be careful about different current working directory

1.9.4. file_put_contents

  • Identical to calling fopen(), fwrite(), fclose() successively to write data to a file

  • Syntax 

    file_put_contents(
string $filename,
mixed $data,
int $flags = 0,
?resource $context = null
): int|false
  • Parameters
    filename
    If it does not exist, the file will be created unless FILE_APPEND flag is set
    data
    a string, an array or a stream resource
    (no term)
    flags
    • FILE_USE_INCLUDE_PATH
    • FILE_APPEND
    • LOCK_EX
  • number of bytes or false on failure
  • file_put_contents('path/to/file', '');

1.10. Module, Extension

1.10.1. bz2

1.10.2. bcmath

1.10.3. calendar

1.10.4. enchant

1.10.5. exif

1.10.6. gd

XPM Suport
libXpm
(no term)
php:gd

1.10.7. gettext

1.10.8. intl

require debian pkg libicu-dev

1.10.9. memcached

  • Port 11211

  • Install 

    sudo apt install memcached php-memcached -y
sudo service apche2 restart

# test
echo "stats" | netcat localhost 11211

# get PHP config for memcached
php -i | grep memcached

# config file
cat /etc/memcached.conf

# restart memcached to clear cache
sudo service memcached restart
  • There's an old version of PHP client which has a different name memcache. On Linux, server/daemon is still memcached

1.10.10. pspell

1.10.11. soap

require debian pkg libxml2-dev

1.10.12. sockets

1.10.13. tidy

1.10.14. xmlrpc

1.11. Locale

In Linux, you can run locale -a to get a list of codes Linux return multi-letter codes. You can use 3-letter or 2-letter codes as a backup. ISO 639 3-letter codes

1.12. Date

1.12.1. date( string $format [, int $timestamp = time() ] ) : string

  • Date formats
  • wp:post:post_date_gmt uses 'Y-m-d\TH:i:sP'">2019-07-16T13:46:16+00:00
  • date function is related to timezone. Use gmdate() for UTC timezone
// Y => 1999
// y => 99 or 03

// m => 01 for January
// n => 1 for January
// M => Jan for January

// d => 01 to 31
// j => 1 to 31

// w => 0 to 6, 0 for Sunday and 6 for Saturday

// H :: 24-hour format for an hour with leading zeros
// G :: 24-hour format of an hour without leading zeros
// h :: 12-hour format of an hour with leading zeros
// g :: 12-hour format of an hour without leading zeros

// i :: minutes with leading zeros

// s :: seconds with leading zeros

// v :: milliseconds without leading zeros. Does not work in `date()` but in `DateTime::format()`

// a :: am or pm
// A :: AM or PM

// Use / to escape
echo date('M j, Y') // Nov 1, 2020 for November 1, 2020
echo date('F j, Y'); // November 1, 2020
echo date('Ymd'); // 20190131 for Jan 31, 2019
echo date('YmdHis'); // 20190131015959 for Jan 31, 2019 01:59:59
echo date('c'); // 2019-05-24T07:11:55-07:00
echo date('y/n/j-G:i:s'); // short but unique

1.12.2. Create a custom date

1.12.2.1. Set Timezone
Supported timezones
https://www.php.net/manual/en/timezones.php
(no term)
US unique timezones
  • https://stackoverflow.com/questions/4989209/list-of-us-time-zones-for-php-to-use
    Eastern Time
    America/New_York == America/Toronto
    Central Time
    America/Chicago == America/Winnipeg
    Mountain Time
    America/Denver == America/Edmonton
    Mountain Time (no DST)
    America/Phoenix
    Pacific Time
    America/Los_Angeles == America/Vancouver
    Alaska Time
    America/Anchorage
    Hawaii-Aleutian
    America/Adak
    Hawaii-Aleutian Time (no DST)
    Pacific/Honolulu
date_default_timezone_set('UTC'); // set the timezone to UTC
date_default_timezone_set('America/Toronto');
date_default_timezone_set('America/New_York');
1.12.2.2. mktime return Unix timestamp
  • mktime() combined with date is very slow. Consider to use strtotime()
// hour, min, sec, month, day, year, $is_dst(daylight saving, 0 or 1) based on the current set timezone
date_default_timezone_set('UTC'); // set the timezone to UTC
$_u_time = mktime(11, 0, 0, 11, 22, 2016);

date_default_timezone_set('America/Toronto');
$timeInTorontoTimeZone = mktime(11, 0, 0, 11, 22, 2016);

$timeInUTCTimeZone = gmmktime(11, 0, 0, 11, 22, 2016);
$_u_time === $timeInUTCTimeZone;
1.12.2.3. strtotime( string $time [, int $now = time() ] ) : int
  • Return Unix timestamp
$_u_time = strtotime('2016-11-01');
// 20190228 format also works

// ISO date string (ISO 8601). See js:time:iso
// Z stands for UTC timezone
$_u_time = strtotime('2011-10-05T14:48:00.000Z');

$dateStr1 = '2022-12-01 00:00:00';
$dateStr2 = '2022-12-01';
$timezone = 'America/Toronto';
var_dump(strtotime($dateStr1 . ' '. $timezone) === 1669870800);
var_dump(strtotime($dateStr2 . ' '. $timezone) === 1669870800);

// Compare
if($start > $end) {}

// difference in seconds
echo $end - $start;

// Format differences

// convert seconds to DateTime object
$dt_start = new DateTime(date('Y-m-d H:i:s', $start));
$dt_end = new DateTime(date('Y-m-d H:i:s', $end));

$interval = date_diff($dt_end, $dt_start);
echo $interval->format('%R%a days');
echo $interval->y; // years

// yesterday
$hour = 12;

$today              = strtotime($hour . ':00:00');
$yesterday          = strtotime('-1 day', $today);
$dayBeforeYesterday = strtotime('-1 day', $yesterday);

$startOfADay = strtotime('today', $anyUnixTimeStamp);
$startOfADay == mktime(0, 0, 0, ...explode('-', date('n-j-Y', $anyUnixTimeStamp))

// strtotime('first day of last month')

echo date('Ymd', $today).PHP_EOL;
echo date('Ymd', $yesterday).PHP_EOL;
echo date('Ymd', $dayBeforeYesterday).PHP_EOL;
1.12.2.4. DateTime, DateInterval

Create by using Unix timestamp (int) in seconds 

$_u_time = strtotime('2016-11-01'); // 20190228 also works

$a = new DateTime(); // now
$a->setTimestamp($_u_time); // set it based on a Unix timestamp to current set timezone
$a->getTimestamp(); // convert back to integer Unix timestamp

$now = new DateTime();

$interval = $now->diff($a);
echo $interval->format('%R%a days');

$yesterday = new DateTime(); // now
$yesterday->sub(new DateInterval('P1D')); // subtract one day
echo $yesterday->format('Ymd');

$myDate = '02111981 00:00:00';
/** @var DateTime|false $date */
$newDate = DateTime::createFromFormat('mdY H:i:s', $myDate); // Current set timezone
$newDate = DateTime::createFromFormat('mdY H:i:s', $myDate, new DateTimeZone('UTC')); // At a timezone other than the current set timezone
$newDateString = ($date) ?
    $date->format('Y-n-j') // error if $date is false
    : false;
$newDateUnix = ($date) ? $date->getTimestamp() : null;

Convert a string format to another 

// to convert a more complex string format to date
// return false if it is not of that format
// http://php.net/manual/en/datetime.createfromformat.php
$date = DateTime::createFromFormat( 'j-M-Y', '15-Feb-2009');

$date_display = $date->format('Y-m-d');

1.12.3. Unix to Format String

$a = strtotime('20190228');
echo gmdate("Y-m-d", $a);

1.12.4. Date in Other Languages (Locale)

$_u_time = mktime (11, 0, 0, 11, 22, 2016);
setlocale(LC_TIME, 'fr_FR.utf8','fra'); // try fr_FR.utf8 first, if fail, try fra
$_fr_time = array();
$_fr_time[] = strftime('%A', $_u_time); // format a date/time according to locale settings
$_fr_time[] = trim(strftime('%e', $_u_time)); // day of month, single digit with leading space if it's single digit..
$_fr_time[] = strftime('%B', $_u_time);
$_fr_time[] = strftime('%Y', $_u_time);
$_fr_time = ucwords(implode(' ', $_fr_time));
echo $_fr_time;

1.12.5. Change timezone

$timezone = date_default_timezone_get(); // UTC
var_dump($timezone); // UTC

var_dump(date('Y/m/d h:i:s a',time())); // e.g. 7pm

date_default_timezone_set('America/Toronto');

var_dump(date('Y/m/d h:i:s a',time())); // e.g. 7pm +4 = 11pm
date_default_timezone_set($timezone); // UTC
var_dump(date('Y/m/d h:i:s a',time())); // e.g. 7pm

1.13. Bitwise Operator

:PROPERTIES: :ID: 3E3C7306-E75A-46ED-83DA-989FB774944D :CUSTOM_ID: php:bitwise END: 

// $a << $b is to shift the bits of $a to left for $b steps (each step multiplyies by two)

define('FLAG_A', 1); /// 0001 or 1<<0
define('FLAG_B', 2); /// 0010 or 1<<1
define('FLAG_C', 4); /// 0100 or 1<<2
define('FLAG_D', 8); /// 1000 or 1<<3

// $a >> $b is to shift the bits of $a to right for $b steps (each step divides by two)
var_dump(8 >> 3 === 1); // 1000 to 1

// Allocated memory in megabytes: becaues 1024 * 1024 = 2^10 * 2*^10 = 2^20 bytes equals to 1 megabyte
$memory = memory_get_usage(true) >> 20;

// 64-bit PHP
var_dump(PHP_INT_MAX === 9223372036854775807);
// 62 is the maximum
var_dump(1<<62 === 4611686018427387904);

$combined_flags = FLAG_A | FLAG_C; // 0101 binary, which is decimal 5
var_dump(decbin($combined_flags) === '101');

var_dump(($combined_flags & FLAG_C) === FLAG_C); // FLAG_C which is 0100, which is decimal 4
// 0101 :: FLAG_A or FLAG_C
// 0100 :: FLAG_C
// 0100 :: result is decimal 4

if ($combined_flags & FLAG_C) {
    var_dump('Has FLAG_C assuming FLAG_C is 1 >> n');
}

$combined_flags2 = $combined_flags | 0;
var_dump( $combined_flags2 === $combined_flags);

var_dump( $combined_flags ^ FLAG_C ); // XOR :: like | but not both
// 0101 :: FLAG_A or FLAG_C
// 0100 :: FLAG_C
// 0001 :: result is decimal 1

1.14. Ternary Operator, Null Coalescing Operator, Elvis Operator, Short if 

// null coalescing operator (??) since PHP 7.0
// It returns its first operand if it exists and is not NULL; otherwise it returns its second operand.

// eq.
$foo = $bar['id'] ?? 'something';
$foo = isset( $bar['id'] ) ? $bar['id'] : 'something';

// Can be used to assign the first non null value. If all values are null, null is used
$foo = $a ?? $b ?? $c;

// Can be used to var_dump to debug
$foo = [
    'a' => $longExpression
];

$foo = [
    'a' => var_dump('description', $longExpression = 'xxx') ?? $longExpression
];

// Elvis Operator (?: a short form of `expression1 ? expression2 : expression3`)
$foo = isset($bar['id']) ?: 'something';
// eq.
$foo = (isset( $bar['id'])) ? (isset($bar['id'])) : 'something';

$bar = abc(); // $bar is always set and it could be 0 or some other non-null values
$foo = $bar ?: 'null'; // non-true values are converted to 'null' and other true values are kept
// eq.
$foo = ($bar) ? $bar : 'null';
// Elvis Operator still throws exception if there's undefined variables

// Since 7.4
$a['abc'] ??= 1;
// eq.
$a['abc'] = $a['abc'] ?? 1;


// PHP 8 null safe operator can work with methods
// Instead of writing this
$country = null;
if ($session !== null) {
    $user = $session->user;
    if ($user !== null) {
        $address = $user->getAddress();

        if ($address !== null) {
            $country = $address->country;
        }
    }
}

// write this
$country = $session?->user?->getAddress()?->country;

// PHP 7.3 we can still use `??` but it only works for properties
var_dump($a->aProperty ?? 'not defined');

// This does not work
// var_dump($a->aMethod() ?? 'not defined');

1.14.1. Short if 

function a() {
    return [ 'a' => '123' ];
}

// standalone with one condition. No parentheses needed
if ( $b = a() ) {
    print_r( $b, 1 );
}

// use parentheses for multiple conditions
if ( ( $b = a() ) && ( '321' !== $c = a() ) ) {}

// a useful example
if ( is_callable( 'a' ) && ( $b = a() ) && isset( $b['a'] ) && $b['a'] == '123' ) {}

// $b is available
var_dump($b);

// Combine with ternory
function abc($a) {
    return ($a) ? "Hello {$a}" : '';
}
$a = ($a = abc('')) ? $a : 'No no'; // No no
$a = ($a = abc('abc')) ? $a : 'No no'; // Hello abc

1.15. Short Echo

  • 1.2.3.9 is not required since PHP 5.4.0
<?php
$a = 'abc';
function abc($text) {
    return "running a function that returns $text";
}
?>
<?= $a ?>
<!-- eq. to -->
<?=$a?>
<!-- eq. to -->
<?= $a; ?>
<!-- eq. to -->
<?php echo $a; ?>

<!-- running a funciton that returns hello! -->
<?= abc('hello!') ?>

<!-- abc_123 -->
<?= "{$a}_123" ?>

<!-- 'false' -->
<?= (0===1) ? 'true' : 'false' ?>

<!-- 'true' -->
<?= ($a === 'abc') ? 'true' : 'false' ?>

<?= ($a === 'abc') ? <<<HTML
<span>
  true
</span>
HTML
    : 'false' ?>

1.16. Error Control Operator

  • All PHP versions
  • https://www.php.net/manual/en/language.operators.errorcontrol.php
  • prepend it to variables, function and include calls
    • Don't append to function/class definition and conditional structures e.g. if, foreach
  • It temporarily sets the error reporting level to 0 for the code
    • If an error is triggered, the error handler is still called but just with error level of 0
    • Set and reset INI setting back and forth
  • Drawbacks
    • It's slow
  • Refer to php:ini:error_reporting
$user = [];
$a = @$user['id']; // by default, no error is output and $a is NULL
$a = (int) @$user['id']; // int 0

1.17. Spaceship Operator

  • Since PHP 7
  • returns -1 ($a less than $b), 0 or 1

1.18. Multibyte php:multibyte

All functions
https://www.php.net/manual/en/ref.mbstring.php
(no term)
Functions mb_* e.g. mb_strlen( 'abc' ) is mb version of strlen
(no term)
str_replace() works with multibyte already
(no term)
Functions work for mb only
php:mb_strimwidth

php:ellipsis mb_strimwidth ( string $str , int $start , int $width [, string $trimmarker = "" [, string $encoding = mb_internal_encoding() ]] ) : string 

echo mb_strimwidth(get_the_title(), 0, 50, '...');

1.18.1. Remove 4 byte characters php:4byte

  • PHP UTF-8 character range is wider than MySQL utf8. MySQL's true UTF-8 is utf8mb4 but sometimes it's hard to change

Here's how to remove any 4 byte character: 

function replace4byte( $string ) {
  return preg_replace( '%(?:
      \xF0[\x90-\xBF][\x80-\xBF]{2}      # planes 1-3
    | [\xF1-\xF3][\x80-\xBF]{3}          # planes 4-15
    | \xF4[\x80-\x8F][\x80-\xBF]{2}      # plane 16
    )%xs', '', $string );
}

var_dump( 
  replace4byte( 'd' ), 
  replace4byte( 'd_Emoji_d' ) 
);

1.19. Array

1.19.1. Spread Operator / Splat

// Spread Operator in Array Expression (Splat Operator) since PHP 7.4
$arr1 = [1, 2, 3];
$arr2 = [
    'a' => 123,
    'b' => array(
        'c' => 123
    ),
];
var_dump([...$arr1] === [1, 2, 3]);
var_dump(array(...$arr1) === [1, 2, 3]);
var_dump([0, ...$arr1] === [0, 1, 2, 3]);
var_dump(array(0, ...$arr1) === [0, 1, 2, 3]);
var_dump([...$arr1, 0] === [1, 2, 3, 0]);
// Fatal error: Unpacking arrays with string keys is forbidden for PHP < 8.1
// var_dump([...$arr2, 0] === [1, 2, 3, 0]);
// Maybe eq. to the following for PHP < 8.1
var_dump(array_merge($arr2, array('c' => 321)) === ['a' => 123, 'b' => ['c' => 123], 'c' => 321]);

// Argument Unpacking and Variadic Functions since 5.6
// - Applies to array or Traversable
$args1 = [1, 2, 3];
$args2 = [4, 5, 6];

// $args is called a variadic parameter
function test(...$args) {
    return $args;
}

var_dump(test(1, 2, 3) === $args1);
var_dump(test(...[1, 2, 3]) === $args1);
var_dump(test(...$args1) === $args1);
var_dump(test(...new ArrayIterator($args1)) === $args1);

var_dump(test(...$args1, ...$args2) === array_merge($args1, $args2));
var_dump(test(1, 2, 3, ...$args2) === array_merge($args1, $args2));

var_dump(test(...array_map('htmlspecialchars', $args)));

// Not possible to use normal arguments after argument unpacking was used. These are invalid
// test(...$args1, 4, 5, 6);
// test(...$args1, 4, 5, 6, ...$args2);

1.19.2. list, compact, extract

$abNumeric = array('abc', 'xyz');
$abAssociative = array(
    'a' => 'abc',
    'b' => 'xyz',
);

var_dump((list($a, $b) = $abNumeric) === $abNumeric);
var_dump($a === 'abc');

list($a, $b) = $abAssociative;
var_dump($a === null);

list($a, $b) = array_values($abAssociative);
var_dump($a === 'abc');

// No warning or notice is thrown. Good!
list($a, $b) = null;
var_dump($a === null, $b === null);

// PHP 7.1.0+
// Order of items in list for associative array does not matter, but don't repeat the key
$listReturn = list('a' => $a, 'b' => $b) = $abAssociative;
var_dump($a === 'abc');

var_dump($listReturn === $abAssociative);

$a = ['a' => 'abc', 'c' => 123];
var_dump(list('c' => $r1, 'b' => $r2, 'a' => $r3) = $a);
// Notice is thrown as $a['c'] cannot be found
var_dump($r1 === 123, $r2 === null, $r3 === 'abc');

list('a' => $a, 'b' => $b, 'e' => $e) = $abcde;
var_dump(compact('a', 'b', 'e'));

$a = 'abc';
$b = 'xyz';

var_dump(compact('a', 'b') === $abAssociative);

// extract(array &$array, int $flags = EXTR_OVERWRITE, string $prefix = ""): int

// Returns the number of variables successfully imported into the symbol table.

$size = "large";
$var_array = array(
    "color" => "blue",
    "size" => "medium",
    "shape" => "sphere",
);
extract($var_array, EXTR_PREFIX_SAME, "wddx");

var_dump($color === 'blue', $size === 'large', $shape === 'sphere', $wddx_size === 'medium');

1.19.3. array_merge, array_merge_recursive php:array_merge

  • array_merge 

    $a = array(
    321 => array('a' => 1, 'b' => 2),
);
$b = array(
    123 => array('a' => 3, 'b' => 4),
);
$c = array(
    123 => array('a' => 5, 'b' => 6),
    987 => array('a' => 7, 'b' => 8),
);

var_dump(
    $test = array_merge($a, $b, $c),
    $test == array(
        0 => array('a' => 1, 'b' => 2),
        1 => array('a' => 3, 'b' => 4),
        2 => array('a' => 5, 'b' => 6),
        3 => array('a' => 7, 'b' => 8),
    )
);

// Use array_replace or + for numeric array
var_dump(
    $test = array_replace($a, $b, $c),
    $test === array(
        321 => array('a' => 1, 'b' => 2),
        123 => array('a' => 5, 'b' => 6),
        987 => array('a' => 7, 'b' => 8),
    )
);

var_dump(
    $test = $a + $b + $c,
    $test === array(
        321 => array('a' => 1, 'b' => 2),
        123 => array('a' => 3, 'b' => 4),
        987 => array('a' => 7, 'b' => 8),
    )
);

$a = array(
    'a' => array('a' => 1, 'b' => 2),
);
$b = array(
    'b' => array('a' => 3, 'b' => 4),
);
$c = array(
    'b' => array('a' => 5, 'b' => 6),
    'c' => array('a' => 7, 'b' => 8),
);

var_dump(
    $test = array_merge($a, $b, $c),
    $test == array(
        'a' => array('a' => 1, 'b' => 2),
        'b' => array('a' => 5, 'b' => 6),
        'c' => array('a' => 7, 'b' => 8),
    )
);

// array_replace is equivalent to array_merge for string-keyed array
var_dump(
    $test = array_replace($a, $b, $c),
    $test === array(
        321 => array('a' => 1, 'b' => 2),
        123 => array('a' => 5, 'b' => 6),
        987 => array('a' => 7, 'b' => 8),
    )
);

var_dump(
    $test = $a + $b + $c,
    $test == array(
        'a' => array('a' => 1, 'b' => 2),
        'b' => array('a' => 3, 'b' => 4),
        'c' => array('a' => 7, 'b' => 8),
    )
);
    • only merge on the first level
    • If key is the same, value at later array will overwrite the first array's value
    • If key does not exist in earlier arrays, the key/value will be appended

    • However, if key is numeric, values will always be appended (not overwriting) disregarding the values are the same

      Use this to dedup for numeric array merging
      array_unique(array_merge($array1,$array2), SORT_REGULAR)
  • array_merge_recursive
    • If key is the same, values will be appended as array
    • Very different from array_merge

1.19.4. Append an array or an element to another array

1.19.4.1. Append an element
$a = array('1', 123);

var_dump(array_push($a, 'hello') === 3);
var_dump(($a[] = 'world') === 'world');
var_dump(
    array_merge($a, array('array_merge')) ===
    array('1', 123, 'hello', 'world', 'array_merge')
);
1.19.4.2. Append several elements array_push($a, 'hello', 'world')
1.19.4.3. Append an array
  • If 2 arrays only have numeric keys e.g. ['a', 1, 2], php:array_merge can be used to append

  • Use this for appending 2 arrays 

    $a = $b = [
['a'=>1,'c'=>321],
['a'=>1, 'b'=>2]
];

$php56Less = $a;
foreach ($b as $item) {
$php56Less[] = $item;
}
var_dump($php56Less, '$php56Less');

$php56AndPlus = $a;
array_push($php56AndPlus, ...$b);
var_dump($php56AndPlus, '$php56AndPlus');

// splat operator since PHP 5.6
$array3 = [
    'prepend',
    ...$array1,
    'inTheMiddle',
    ...$array2,
    'append',
];

1.19.5. Union $array1 + $array2

$a = [
    'one' => 'a1',
    'two' => 'a2',
    'three' => 'a3'
];

$b = [
    'two' => 'b2',
    'three' => 'b3',
    'four' => 'b4'
];

var_dump($a + $b);

$r = [
    'one' => 'a1',
    'two' => 'a2',
    'three' => 'a3',
    'four' => 'b4'
];

1.19.6. Last Element, First Element, First Key

// First element/value
// PHP 5.4+
$first = array_values($array)[0]; // throw error if array is empty

// PHP 7.3+
$first = $array[array_key_first($array)];
// array_key_first($param) requires $param to be array otherwise warning will be thrown
// array_key_first([]) returns null
// Use this
$first = is_array($array)
  ? ($array[array_key_first($array)] ?? "default value if $array is an empty array")
  : "default value if $array is not an array";

// $first = current($array); // requires the array current pointer is at the beginning
// $first = reset($array); // mutates the array (set the current pointer to the first element) and returns the first array element value or false if the array is empty!

// Last element
$last = end($target_array); // set array pointer to last
$last = reset($target_array);
// PHP 7.3+
(is_array($array) && $array) ? $array[array_key_last($array)] : 'variable is empty or invalid';

1.19.7. array_shift, array_unshift prepend elements, array_pop

// array_pop ( array &$array ) : mixed
// pop and return the value of the last element of array, shorten the array by one element

// array_shift ( array &$array ) : mixed
// first element, literal keys won't be affected

// array_unshift ( array &$array [, mixed $... ] ) : int
// prepend passed elements to the front of the array. Literal keys won't be changed

$queue = ['orange', 'banana'];
$queue2 = $queue;
array_unshift($queue, 'apple', 'raspberry'); // ['apple', 'raspberry', 'orange', 'banana']
// to prepend into associative array, use array union (+) or array_merge

$toAdd = ['apple'];
// no mutation using array_merge for numeric arrays
$queue2 = array_merge($toAdd, $queue2); // ['apple', 'orange', 'banana']

1.19.8. in_array

in_array ( mixed $needle , array $haystack [, bool $strict = FALSE ] ) : bool
  • if needle is a string and the array is string[], then case-sensitive comparison
  • Loose comparison unless $strict is set
  • also check the types of the needle in the haystack
var_dump(in_array(null, null) === null);
var_dump(in_array(3, null) === null);

var_dump(in_array(null, [0, 1]) === true);

var_dump(in_array('', [0, 1]) === true);

var_dump(in_array('1', [0, 1]) === true);
var_dump(in_array('3', [0, 1]) === false);
var_dump(in_array('1', [0, 1], true) === false);

// best solution
var_dump(in_array(intval('3'), [0, 1]) === false);

1.19.9. Search by a value and return keys

  • Case insensitive search 

    $a= array(
'k1'=> 'one',
'k2' => 'one1',
'k3' => 'three',
);

print_r( preg_grep( "/ONe$/i" , $a ) );
// return all matches
// Array ( [k1] => one, ... )
// empty array if no match
  • Use in_array for case sensitive search

1.19.10. Search by keys and return values in order

function arrayFilterByKeysInOrder($array, $keys, $default = null, $valuesOnly = true, $notToSetIfNotExists = false) {
    $allKeys = array_keys($array);
    $array2 = [];
    if ($notToSetIfNotExists) {
        return array_intersect_key($array, array_flip($keys));
    }
    foreach ($keys as $k) {
        $r = (in_array($k, $allKeys)) ? $array[$k] : $default;
        if ($valuesOnly) {
            $array2[] = $r;
        }
        else {
            $array2[$k] = $r;
        }
    }

    return $array2;
}
$a = [
    'a' => 123,
    'b' => 'abc',
    'c' => 321,
];
var_dump(
    arrayFilterByKeysInOrder($a, ['c', 'abc', 'a'])  == [
        321,
        null,
        123
    ]
);
var_dump(
    arrayFilterByKeysInOrder($a, ['c', 'abc', 'a'], null, false)  == [
        'c' => 321,
        'abc' => null,
        'a' => 123
    ]
);
var_dump(
    arrayFilterByKeysInOrder($a, ['c', 'abc', 'a'], null, false, true)  == [
        'c' => 321,
        'a' => 123,
    ]
);

1.19.11. Search by value and return the first key by value array_search 

array_search ( mixed $needle , array $haystack [, bool $strict = FALSE ] ) : mixed
return
the first key that has the value or false
(no term)
Case-insensitve
(no term)
Examples

  • Search value in a multidimensional array 

    $db = array(
    array('uid' => '100'),
    array('uid' => 100, 'optional' => 123),
);

// PHP 5.5+ to support array_column
var_dump(array_search('100', array_column($db, 'uid')) === 0);
var_dump(array_search(100, array_column($db, 'uid')) === 0);
// every item in $db has to have the column/array key as uid, otherwise, array_search returns the index of the first row that does not have the column/array key
var_dump(array_search(123, array_column($db, 'optional')) === 0);

// PHP < 5.5
var_dump(array_search('100', array_map(function ($v) {
        return $v['uid'];
    }, $db)) === 0);
var_dump(array_search(100, array_map(function ($v) {
        return $v['uid'];
    }, $db)) === 0);
var_dump(array_search(123, array_map(function ($v) {
        return $v['optional'];
    }, $db)) === 1);
  • $key = array_search('100', array_column($userdb, 'uid')); Since PHP 5.5
    • every item in $userdb has to have the column/array key as uid, otherwise, array_search returns the index of the first row that does not have the column/array key

1.19.12. Delete by a value

  • Case-sensitive
foreach (array_keys($items, 'search this value', true) as $key) {
    // true is case-insensitive
    unset($items[$key]);
}

// Comparison is done by `(string) $elem1 === (string) $elem2`
$items = array_diff($items, ['search this value']);

// PHP 7.4
$items = array_filter($items, fn ($m) => $m != 'search this value');

1.19.13. Change Case for All Keys 

$_lower_GET = array_change_key_case($_GET, CASE_LOWER); // Or CASE_UPPER

1.19.14. Change keys only

  • The length of $keys and $values should be equal
array_combine(array $keys, array $values): array

$numeric_array = array(1, 2, 3);
$assign_keys = array('a', 'b', 'c');
array_combine($assign_keys, $numeric_array) === array (
  'a' => 1,
  'b' => 2,
  'c' => 3,
);

$a = array(
    'a_321' => '123',
    'abc' => '321',
);

// Uppercase the first letter in keys
var_export($test = array_combine(array_map('ucfirst', array_keys($a)), $a));
var_dump(
    $test === array(
        'A_321' => '123',
        'Abc' => '321',
    )
);

// Remove _ in keys
var_export(
    $test = array_combine(
        array_map(
            function ($item) {
                return str_replace('_', '', $item);
            },
            array_keys($a)
        ),
        $a
    )
);
var_dump(
    $test === array(
        'a321' => '123',
        'abc' => '321',
    )
);

$indexDBRows = array(
    0 => array('id' => 123, 'name' => 'Li'),
    1 => array('id' => 124, 'name' => 'Hello'),
    2 => array('id' => 125, 'name' => 'World'),
);

$primaryKeyDBRows = array(
    123 =>
        array(
            'id' => 123,
            'name' => 'Li',
        ),
    124 =>
        array(
            'id' => 124,
            'name' => 'Hello',
        ),
    125 =>
        array(
            'id' => 125,
            'name' => 'World',
        ),
);

// PHP 5.5+
$r = array_combine(array_column($indexDBRows, 'id'), $indexDBRows);
var_dump($r === $primaryKeyDBRows);

// PHP < 5.5
$r = array_combine(
    array_map(function($v) { return $v['id']; }, $indexDBRows),
    $indexDBRows
);
var_dump($r === $primaryKeyDBRows);
1.19.14.1. Convert Numeric Arrays to Associative Arrays
function arrayNumericToAssociative(array $keys, array $array, $options = array()): array {
    array_walk(
        $keys,
        function ($v, $k) use (&$r, $array, $options) {
            if (!isset($v)) {
                return $r;
            }
            $r[$v] = array_key_exists($k, $array)
                ? $array[$k]
                : ($options['default'] ?? null);
            return true;
        },
        $r = array()
    );
    return $r;
}

$keys = array('a', 'b', 'c');
arrayNumericToAssociative($keys, [1, 2, 3, 4]) === ['a' => 1, 'b' => 2, 'c' => 3];
arrayNumericToAssociative($keys, [1, 2]) === ['a' => 1, 'b' => 2, 'c' => null];
arrayNumericToAssociative($keys, []) === ['a' => null, 'b' => null, 'c' => null];

1.19.15. Get values of a column in a multidimensional array

Since PHP 5.5 

$records = [
    [
        'id' => 2135,
        'first_name' => 'John',
        'last_name' => 'Doe',
    ],
    [
        'id' => 3245,
        'first_name' => 'Sally',
        'last_name' => 'Smith',
    ],
    [
        'id' => 5342,
        'first_name' => 'Jane',
        // Notice! there's no last_name
    ],
    [
        'id' => 5623,
        'first_name' => 'Peter',
        'last_name' => 'Doe',
    ],
];

$a1 = array_column($records, 'last_name'); // ['Doe', 'Smith', 'Doe']
$a2 = array_column($records, 'last_name', 'id'); // [2135 => 'Doe', 3245 => 'Smith', 5623 => 'Doe']

array_column([], 'id') === [];
array_column(null, 'id') === null;

// no __get and __isset() magic methods
class User {
    public $username;

    public function __construct(string $username) {
        $this->username = $username;
    }
}

$users = [
    new User('user 1'),
    new User('user 2'),
    new User('user 3'),
];

$a3 = array_column($users, 'username'); // ['user 1', 'user 2', 'user 3']

// __get and __isset() magic methods and get private properties
class Person {
    private $name;

    public function __construct(string $name) {
        $this->name = $name;
    }

    public function __get($prop) {
        return $this->$prop;
    }

    public function __isset($prop): bool {
        return isset($this->$prop);
    }
}

$people = [
    new Person('Fred'),
    new Person('Jane'),
    new Person('John'),
];

$a4 = array_column($people, 'name'); // ['Fred', 'Jane', 'John']

1.19.16. array_diff, array_diff_key, array_diff_assoc, array_intersect, array_intersect_assoc

  • Use array_diff, array_diff_key, array_intersect for numeric index array
  • Use array_diff_assoc, array_intersect_assoc for associative array
array_diff(array $array, array ...$arrays): array
  • Compare on values, and return the key/value pairs in the first array for the values don't exist in the following array(s)
  • Two elements are considered equal if and only if (string) $elem1 === (string) $elem2
  • Throw Notice when element cannot be cast to a string
  • Can be used to remove elements of $arraySource that are in $arrayValuesToRemove
    To keep original index
    array_diff($arraySource = [312, 401, 15, 401, 3], $arrayValuesToRemove = [401]) === [0 => 312, 2 => 15, 4 => 3]
    Continuous index
    array_values(array_diff([312, 401, 15, 401, 3], [401])) === [312, 15, 3]
array_intersect(array $array, array ...$arrays): array
array_intersect_assoc(array $array, array ...$arrays): array
  • Return an array containing all the values of the first array that are present in all other arrays
array_intersect_key(array $array, array ...$arrays): array
  • Computes the intersection of arrays using keys for comparison

  • Filter an array by keys 

    $my_array = array("foo" => 1, "hello" => "world");

$allowed = array("foo", "bar");

array_intersect_key($my_array, array_flip($allowed)) === array("foo" => 1);

Single dimension 

$arr = ['nice_item', 'remove_me', 'another_liked_item', 'remove_me', 'remove_me_also'];
$arr = array_diff($arr, ['remove_me', 'remove_me_also']); // [ 0 => 'nice_item', 2 => 'another_liked_item']

$hasKeysThatAreNotSupposedToHave = ['nice_item', 'new key not existing'];
$diff = array_diff($hasKeysThatAreNotSupposedToHave, $arr); // [ 1 => 'new key not existing' ]

// Check if $arr has all these keys
$hasAllKeys = ['nice_item', 'another_liked_item', 'another_liked_item']; // keys can be repeated
$missingKeys = array_diff($hasAllKeys, $arr);
if (empty($missingKeys)) {
    echo 'Has all keys';
} else {
  var_dump($missingKeys, 'These keys are missing');
}

Associative arrays 

$array1 = array('blue' => 1, 'red' => 2, 'green' => 3, 'purple' => 4);
$array2 = array('green' => 5, 'yellow' => 7, 'cyan' => 8);
$array3 = array('green' => 3, 'yellow' => 7, 'cyan' => 8);

var_export(
    array_diff($array1, $array2) === array(
        'blue' => 1,
        'red' => 2,
        'green' => 3,
        'purple' => 4,
    )
);

var_export(
    array_diff($array1, $array3) === array(
        'blue' => 1,
        'red' => 2,
        'purple' => 4,
    )
);

var_export(
    array_diff_key($array1, $array2) === array(
        'blue' => 1,
        'red' => 2,
        'purple' => 4,
    )
);
1.19.16.1. array_diff_key - Compare on keys
array_diff_key(array $array, array ...$arrays): array

$values = [
    'a' => 123,
    'b' => 321,
    'c' => 'abc',
];

$keysWithDummyValues = array_fill_keys(['a', 'z', 'c'], 0);
var_dump($keysWithDummyValues === ['a' => 0, 'c' => 0, 'z' => 0]);
var_dump(array_diff_key($values, $keysWithDummyValues) === ['b' => 321]);
var_dump(array_diff_key($values, []) === $values);
var_dump(array_diff_key($values, null) === false);
$keysWithDummyValues2 = array_fill_keys(['x', 'y', 'b'], 0);
var_dump(array_diff_key($values, $keysWithDummyValues2) === ['a' => 123, 'c' => 'abc']);

1.19.17. array_key_exists 

function array_key_exists($key, array $search) bool 

$a = array(
    'a' => 123,
    'b'=> 321,
);

// Check if a single key exists in an array
var_export(array_key_exists('a', $a) === true);

// Check if multiple keys all exist in an array
function arrayKeysAllExist($keys, $arr, $onlyThoseKeys = false): bool {
    if ($onlyThoseKeys) {
        $arrKeys = array_keys($arr);
        $diff1 = array_diff($keys, $arrKeys);
        return $diff1 == array() && array_diff($arrKeys, $keys) == $diff1;
    }
    return !array_diff_key(array_flip($keys), $arr);
}
var_dump(arrayKeysAllExist(array('a', 'c'), $a) === false);
var_dump(arrayKeysAllExist(array('a'), $a) === true);
var_dump(arrayKeysAllExist(array('a', 'a'), $a) === true);
var_dump(arrayKeysAllExist(array('a', 'a'), $a, true) === false);

function array_keys_exist_count(array $keys, array $arr) {
    return count(array_intersect_key(array_flip($keys), $arr));
}
var_export(array_keys_exist_count(array('a', 'c'), $a) == 1);

1.19.18. array_map

  • Do something with each array element's value and return a new array
  • See array_combine to change keys
array_map(?callable $callback, array $array, array ...$arrays): array

function sanitize($s) {
  return htmlspecialchars($s);
}
$a = array('harmless', '<bad>', '>>click here!<<');
$a = array_map('sanitize', $a); // immutable
// trim each element
$a = array_map('trim', $a);
echo implode(' ', $a);

// Lambda function is available since PHP 5.3.0
$func = function($value) {
    return $value * 2;
};
print_r(array_map($func, range(1, 5)));

// multiple arrays can be passed
$letters = [
    'a' => 'a-abc',
    'b' => 'b-abc',
];

$numericArray = array_map(
    function ($letter, $index) {
        return "Index at $index is $letter";
    },
    // fn($letter, $index) => "Index at $index is $letter", // arrow function since PHP 7.4
    $letters,
    array_keys($letters)
);

var_dump($numericArray === ['Index at a is a-abc', 'Index at b is b-abc']);

1.19.19. array_splice Insert to any position of an array

1.19.19.1. Example
$original = array('a', 'b', 'c', 'd', 'e');
$inserted = array('x', 'y');

// remove and replace elements in array
// array_splice (array &$input, $offset, $length = null, $replacement = null)
// return the array consisting of the extracted elements.
$extractedElements = array_splice($original, 3, 0, $inserted);

var_dump(
    'Length is 0 means to insert at position 3 (offset)',
    $original === array('a', 'b', 'c', 'x', 'y', 'd', 'e')
);

var_dump(
    'Extracted elements is empty array because only insert',
    $extractedElements === array()
);

array_unshift($original, 'f', 'g');

var_dump(
    'Prepend f and g in front',
    $original === array('f', 'g', 'a', 'b', 'c', 'x', 'y', 'd', 'e')
);

$insertedOne = 'a string';
$extractedElements = array_splice($original, 3, 0, $insertedOne);

var_dump(
    'Inserted One',
    $original === array('f', 'g', 'a', 'a string', 'b', 'c', 'x', 'y', 'd', 'e')
);

$original = array('a' => 1, 'b' => 2, 'c' => 3, 'd' => 4, 'e' => 5);
$inserted = array('x' => 6, 'y' => 7);
$extractedElements = array_splice($original, 3, 0, $inserted);

var_dump(
    'Associative array',
    $original === array(
        'a' => 1,
        'b' => 2,
        'c' => 3,
        0 => 6,
        1 => 7,
        'd' => 4,
        'e' => 5,
    )
);

var_dump(
    'array_keys($original)',
    array_keys($original) === ['a', 'b', 'c', 0, 1, 'd', 'e'],
    'Find Index By Key',
    findIndexByKey($original, 'c') === 2,
    findIndexByKey($original, 'd') === 5,
    findIndexByKey($original, 1) === 4,
    findIndexByKey($original, '1') === false,
    findIndexByKey($original, 0) === 3,
    findIndexByKey($original, '0') === false,
    findIndexByKey(array_merge($original, array('d' => 6)), 'd') === 5,
    findIndexByKey(array('a' => 1, 'b' => 2, 'c' => 3, 'd' => 4, 'e' => 5, 'd' => 6), 'd') === 3
);

var_dump(
    'Insert before key d',
    insertKeyValuePairsToAssocArray($original, 'd', $inserted) === array(
        'a' => 1,
        'b' => 2,
        'c' => 3,
        0 => 6,
        1 => 7,
        'x' => 6,
        'y' => 7,
        'd' => 4,
        'e' => 5,
    ),
    'Insert after key d',
    insertKeyValuePairsToAssocArray($original, 'd', $inserted, false) === array(
        'a' => 1,
        'b' => 2,
        'c' => 3,
        0 => 6,
        1 => 7,
        'd' => 4,
        'x' => 6,
        'y' => 7,
        'e' => 5,
    ),
    'Insert after key e (the last key in original array)',
    insertKeyValuePairsToAssocArray($original, 'e', $inserted, false) === array(
        'a' => 1,
        'b' => 2,
        'c' => 3,
        0 => 6,
        1 => 7,
        'd' => 4,
        'e' => 5,
        'x' => 6,
        'y' => 7,
    ),
    'Insert after key 1',
    insertKeyValuePairsToAssocArray($original, 1, $inserted, false) === array(
        'a' => 1,
        'b' => 2,
        'c' => 3,
        0 => 6,
        1 => 7,
        'x' => 6,
        'y' => 7,
        'd' => 4,
        'e' => 5,
    )
);

function findIndexByKey($array, $key) {
    return array_search($key, array_keys($array), true);
}

var_export(
    insertKeyValuePairsToAssocArray(
        [1, 2, 3, 'x' => 'abc', 'y' => 1, 'z' => null],
        'z', ['xyz' => true],
        false
    ) ===
    [1, 2, 3, 'x' => 'abc', 'y' => 1, 'z' => null, 'xyz' => true]
);

var_export(
    insertKeyValuePairsToAssocArray(
        [1, 2, 3, 'x' => 'abc', 'y' => 1, 'z' => null],
        'z', ['xyz' => true, 'x' => 'override'],
        false
    ) ===
    [1, 2, 3, 'x' => 'override', 'y' => 1, 'z' => null, 'xyz' => true]
);

/**
 * Insert key/value pairs before or after a key/value
 *
 * @param array $array array_merge is used for numeric keys, please use it carefully. If array is purely string-keyed, no problem
 * @param string|int $insertAtThisKey
 * @param array $insertItems
 * @param bool $insertBefore otherwise, insert after
 *
 * @return array
 * @throws Exception
 */
function insertKeyValuePairsToAssocArray($array, $insertAtThisKey, $insertItems, $insertBefore = true): array {
    if (!array_key_exists($insertAtThisKey, $array)) {
        throw new Exception('Key cannot be found.');
    }
    $keyAtIndex = findIndexByKey($array, $insertAtThisKey);
    $indexToInsert = $insertBefore ? $keyAtIndex : $keyAtIndex + 1;
    $arrayEnd = array_splice($array, $indexToInsert);
    $arrayStart = array_splice($array, 0, $indexToInsert);
    return array_merge($arrayStart, $insertItems, $arrayEnd);
}
1.19.19.2. Example not using array_splice
$original = array(
    array(
        'paymentid' => '2255997',
        'subItems' =>
            array(
                array(
                    'paymentid' => '2255995',
                ),
                array(
                    'paymentid' => '2255996',
                ),
            ),
    ),
    array(
        'paymentid' => '2256997',
        'subItems' =>
            array(
                array(
                    'paymentid' => '2256995',
                ),
                array(
                    'paymentid' => '2256996',
                ),
            ),
    ),
);

$result = array(
    array(
        'paymentid' => '2255997',
        'subItems' =>
            array(
                array(
                    'paymentid' => '2255995',
                ),
                array(
                    'paymentid' => '2255996',
                ),
            ),
    ),
    array(
        'paymentid' => '2255995',
    ),
    array(
        'paymentid' => '2255996',
    ),
    array(
        'paymentid' => '2256997',
        'subItems' =>
            array(
                array(
                    'paymentid' => '2256995',
                ),
                array(
                    'paymentid' => '2256996',
                ),
            ),
    ),
    array(
        'paymentid' => '2256995',
    ),
    array(
        'paymentid' => '2256996',
    ),
);

$copy = [];
foreach ($original as $key => $value) {
    $copy[] = $value;
    if (!($value['subItems'] ?? null)) {
        continue;
    }
    $itemsToInsert = $value['subItems'];
    // $copy[$key]['subItems'] = [];
    $copy = array_merge($copy, $itemsToInsert);
}

var_dump($copy === $result);

$original = array(
    2255997 =>
        array(
            'paymentid' => '2255997',
            'subItems' =>
                array(
                    2255995 => array(
                        'paymentid' => '2255995',
                    ),
                    2255996 => array(
                        'paymentid' => '2255996',
                    ),
                ),
        ),
    2256997 =>
        array(
            'paymentid' => '2256997',
            'subItems' =>
                array(
                    2256995 => array(
                        'paymentid' => '2256995',
                    ),
                    2256996 => array(
                        'paymentid' => '2256996',
                    ),
                ),
        ),
);

$result = array(
    2255997 =>
        array(
            'paymentid' => '2255997',
            'subItems' =>
                array(
                    2255995 => array(
                        'paymentid' => '2255995',
                    ),
                    2255996 => array(
                        'paymentid' => '2255996',
                    ),
                ),
        ),
    2255995 => array(
        'paymentid' => '2255995',
    ),
    2255996 => array(
        'paymentid' => '2255996',
    ),
    2256997 =>
        array(
            'paymentid' => '2256997',
            'subItems' =>
                array(
                    2256995 => array(
                        'paymentid' => '2256995',
                    ),
                    2256996 => array(
                        'paymentid' => '2256996',
                    ),
                ),
        ),
    2256995 => array(
        'paymentid' => '2256995',
    ),
    2256996 => array(
        'paymentid' => '2256996',
    ),
);

$copy = [];
foreach ($original as $key => $value) {
    $copy[$key] = $value;
    if (!($value['subItems'] ?? null)) {
        continue;
    }
    foreach ($value['subItems'] as $subKey => $subValue) {
        $copy[$subKey] = $subValue;
    }
}

var_dump($copy === $result);

1.19.20. array_slice Extract a slice of the array

array_slice(
    array $array,
    int $offset,
    ?int $length = null,
    bool $preserve_keys = false
): array

$input = array("a", "b", "c", "d", "e");

$output = array_slice($input, 2);      // returns "c", "d", and "e"
$output = array_slice($input, -2, 1);  // returns "d"
$output = array_slice($input, 0, 3);   // returns "a", "b", and "c"

1.19.21. array_fill fill values

// array_fill ($start_index, $num, $value)
var_dump(
    'Fill from index 0 to 2',
    array_fill(0, 2, 'x') === array('x', 'x')
);

var_dump(
    'Fill from index 1 to 2',
    array_fill(1, 2, 'x') === array(1 => 'x', 2 => 'x')
);

1.19.22. array_fill_keys 

$keys = array('foo', 5, 10, 'bar');
array_fill_keys($keys, 'banana') === array(
    'foo' => 'banana',
    5 => 'banana',
    10 => 'banana',
    'bar' => 'banana',
);

1.19.23. array_walk

Side-effect
may mutate source array values. And only array values can be changed.
  • Cannot change array keys! Use array_combine
    • Cannot add, unset or reorder elements/keys in array
(no term)
Can access source array key and value
  • array_reduce can only access value, but it returns a value after the loop
(no term)
Return bool
  • To return the new array, see array_map
  • May use return in callback for guard clauses
(no term)
Will walk through the entire array regardless of pointer position
(no term)
No break nor continue
array_walk ( array|object &$array , callable $callback , mixed $userdata = null ) : bool

$fruits = array("d" => "lemon", "a" => "orange", "b" => "banana", "c" => "apple");
array_walk($fruits, function($v, $k) {
    printf('%s="%s" ', $k, $v);
});

// To alter $fruits
array_walk($fruits, function(&$v, $k, $prefix) {
    $v = "$prefix: $v";
}, 'customPrefix');
var_dump('$fruits', $fruits);

// Don't do this to alter external param $fruits2 !!!
$fruits2 = [];
array_walk($fruits, function($v, $k, &$externalVar) {
    $externalVar[$k] = $v;
}, $fruits2);
var_dump('$fruits2', $fruits2 === []);

// To alter external params #1
array_walk($fruits, function($v, $k) use (&$fruits2) {
    $fruits2[$k] = $v;
}, $fruits2 = []);
var_dump('$fruits2', $fruits2); // works!

// To alter external params #2
$fruits3 = [];
array_walk($fruits, function($v, $k) use (&$fruits3) {
    $fruits3[$k] = $v;
});
var_dump('$fruits3', $fruits3); // works!

// Other external params can be passed (even as a reference)
$counter = 0;
array_walk($fruits, function($v, $k) use (&$fruits2, &$counter) {
    $fruits2[$k] = $v;
    $counter++;
}, $fruits2 = []);
var_dump('$fruits2', $fruits2, '$counter', $counter); // works!

// Walk an existing array and create a new array $index
array_walk($array, function ($v, $k) use (&$index) {
    $index[] = $k;
}, $index = []);

// Unset a subkey in a multi-dimensional array
// unset subkey recognizedFields in a deep nested array
array_walk($r['results'][$entityTitle]['items'], function ($v, $k) use (&$item) {
    unset($item[$k]['recognizedFields']);
}, $r['results'][$entityTitle]['items']);

1.19.24. array_filter

  • Basics 

    array_filter(array $array, ?callable $callback = null, int $mode = 0): array
  • cb function should return true or false. By default, only value is passed to the callback function. Unless flag is used
  • array_filter($my_array) can be used to roughly filter empty/null elements out

    • array element value == false 

      $simpleXMLElementWithoutChildenNorAttributes = new SimpleXMLElement("<news></news>");
$a = array(
    '0', // empty
    '', // empty
    null, // empty
    0.0, // empty
    -0.0, // empty
    0, // empty
    false, // empty
    array(), // empty
    $simpleXMLElementWithoutChildenNorAttributes, // empty
    ' ',
    -1,
);

var_export(array_diff($a, array_filter($a)) === array(
    0 => '0',
    1 => '',
    2 => NULL,
    4 => 0.0,
    5 => -0.0,
    6 => 0,
    8 => false,
    9 => array(),
    10 => $simpleXMLElementWithoutChildenNorAttributes,
));
  • Original index number is used!
  • See 1.19.16 to filter an array by keys
$a = [
    '1' => null,
    '2' => 0,
    '3' => '0',
    '4' => '',
    '5' => false,
    '6' => 123,
    '7' => -1
];

var_dump(array_filter($a) === array('6' => 123, '7' => -1));

array_filter(explode(',', $string));
array_filter(explode(',', $string), 'strlen');

$arr = ['a' => 1, 'b' => 2, 'c' => 3, 'd' => 4];

// default values are passed. To pass the keys of array, use a flag
var_dump(array_filter($arr, function($k) {
    return $k == 'b';
}, ARRAY_FILTER_USE_KEY));

var_dump(array_filter($arr, function($v, $k) {
    return $k == 'b' || $v == 4;
}, ARRAY_FILTER_USE_BOTH));

$clients = [
    'Li' => 'hello',
];
$htmlOptions = '';
$fn = function ($d = '') {
    return $d;
};
array_filter($clients, function ($v, $k) use (&$htmlOptions, $fn) {
    if (!$k) {
        return false;
    }
    $htmlOptions .= <<<HTML
    <option value="{$fn(htmlspecialchars($k))}">{$fn(htmlspecialchars($v))}</option>
HTML;
    return false;
}, ARRAY_FILTER_USE_BOTH);
var_dump($htmlOptions);

1.19.25. array_reduce

  • See js:array:reduce 

    array_reduce ( array $array , callable $callback [, mixed $initial = NULL ] ) : mixed

callback ( mixed $carry , mixed $item ) : mixed

    $sum = 100;
$fees = array(
    '1' => array('amount' => 1),
    '2' => array('amount' => 2),
);
$r = array_reduce($fees, function($carry, $item) {
    return $carry -= $item['amount'];
}, $sum);
var_dump($r === 97);
  • Return
    • null
      • if $array is not an array and warning is thrown
      • if $array is an empty array and $initial is not set
  • Can be associative array, but no way to access the key in callback function
  • Return a value after loop
    • Could be null if $array is not an array

  • Return deep value in array 

    /**
 * @param array $array
 * @param array|string|int $keys
 * @param null $default
 *
 * @return mixed|null
 */
function arrayGetDeepValue($array, $keys, $default = null) {
    $keys = ( ! is_array($keys)) ? array($keys) : $keys;
    $r = array_reduce(
        $keys,
        function ($result, $index) {
            return (isset($result) && array_key_exists($index, $result)) ? $result[$index] : null;
        },
        $array
    );

    return isset($r) ? $r : $default;
}

1.19.26. array_sum()

function arraySubtract($array) {
    $r = null;
    array_walk(
        $array,
        function ($v) use (&$r) {
            $r = isset($r) ? $r - $v : $v;
        },
        $r
    );

    return $r;
}

var_dump($test = array_sum(array(100, 10, 20, 30)), $test === 100 + 10 + 20 + 30);
var_dump($test = arraySubtract(array(100, 10, 20, 30)), $test === 100 - 10 - 20 - 30);
var_dump($test = arraySubtract(array('a' => 100, 'b' => 10, 'c' => 20, 'd' => 30)), $test === 100 - 10 - 20 - 30);
var_dump($test = array_sum(array('a' => 100, 'b' => 10, 'c' => 20, 'd' => 30)), $test === 100 + 10 + 20 + 30);

1.19.27. Sort arrays

1.19.27.1. sort
  • Sort by value, the sorted array has keys starting from 0 (key/value pair is not maintained)
    sort
    ascending order
    rsort
    descending order
  • Mutation
  • Flags
    • SORT_REGULAR
    • SORT_NUMERIC
    • SORT_STRING
    • SORT_LOCALE_STRING
    • SORT_NATURAL
    • case-insensitive for SORT_STRING and SORT_NATURAL
sort ( array &$array [, int $sort_flags = SORT_REGULAR ] ) : bool

$fruits = array('d' => 'lemon', 'a' => 'orange', 'b' => 'banana', 'c' => 'apple');
sort($fruits);
var_dump($fruits === array('apple', 'banana', 'lemon', 'orange'));
$fruits = array('lemon', 'orange', 'banana', 'apple');
sort($fruits);
var_dump($fruits === array('apple', 'banana', 'lemon', 'orange'));
1.19.27.2. ksort, krsort
  • Sort (associative) array by its key and maintain key to data correlations
    ksort
    ascending order
    krsort
    descending order
  • Mutation
  • Return true on success or false on failure
krsort(array &$array, int $flags = SORT_REGULAR): bool
1.19.27.3. asort, arsort
  • Sort (associative) array by its value and maintain key to data correlations
    asort
    ascending order
    arsort
    descending order
  • Similar to 1.19.27.2
    • When target array is numeric array, the key is maintained
  • Mutation
  • Return true on success or false on failure
arsort(array &$array, int $flags = SORT_REGULAR): bool

$sortedKeyValuePair = array (
    'a' => 'orange',
    'd' => 'lemon',
    'b' => 'banana',
    'c' => 'apple',
);
$fruits = array('d' => 'lemon', 'a' => 'orange', 'b' => 'banana', 'c' => 'apple');
arsort($fruits);
var_dump($fruits, $fruits === $sortedKeyValuePair);
1.19.27.4. usort, uasort - custom sort by value
  • Mutation
  • Sort by value
  • usort doesn't maintain key association but uasort does. If callback function returns negative integer, means the comparison is less than
  • See Spaceship operator
  1. Basic example
    $a = array(
    array('id' => 321, 'name' => 'Li Li 321'),
);
$b = array(
    array('id' => 456, 'name' => 'Li Li 456'),
    array('id' => 123, 'name' => 'Li Li 123'),
);
$c = array(); // no warning
$d = array(
    array('name' => 'Li Li Random no id'), // PHP Notice: no index 'id'
);

$toSort = array_merge($a, $b);
$toSort2 = array_merge($b, $c);
$toSort3 = array_merge($b, $d);

function mySort($a, $b) {
    return $a['id'] - $b['id']; // ascending order
}

usort($toSort, 'mySort');
var_export($toSort);
var_dump(
    $toSort === array(
        array('id' => 123, 'name' => 'Li Li 123',),
        array('id' => 321, 'name' => 'Li Li 321',),
        array('id' => 456, 'name' => 'Li Li 456',),
    )
);
usort($toSort2, 'mySort');
var_export($toSort2);
var_dump(
    $toSort2 === array(
        array('id' => 123, 'name' => 'Li Li 123'),
        array('id' => 456, 'name' => 'Li Li 456'),
    )
);
usort($toSort3, 'mySort');
var_export($toSort3);
var_dump(
    $toSort3 === array(
        array('name' => 'Li Li Random no id'), // PHP Notice: no index 'id'
        array('id' => 123, 'name' => 'Li Li 123'),
        array('id' => 456, 'name' => 'Li Li 456'),
    )
);
  2. More examples
    $to_sort = [
    [
        'hashtag' => 'a7e87329b5eab8578f4f1098a152d6f4',
        'title' => 'Flower',
        'order' => 3
    ],
    ['hashtag' => 'b24ce0cd392a5b0b8dedc66c25213594',
     'title' => 'Free',
     'order' => 2
    ],
    [
        'hashtag' => 'e7d31fc0602fb2ede144d18cdffd816b',
        'title' => 'Ready',
        'order' => 1
    ]
];

// PHP 5.2 and lower
function sortByOrder($a, $b) {
    return $a['order'] - $b['order']; // ascending order
}
usort($myArray, 'sortByOrder');

// Since PHP 5.3
usort($myArray, function($a, $b) {
    return $a['order'] - $b['order'];
});

// Since PHP 7
usort($myArray, function($a, $b) {
    return $a['order'] <=> $b['order'];
});

// Order by multiple values
usort($myArray, function($a, $b) {
    $retval = $a['order'] <=> $b['order'];
    if ($retval == 0) {
        $retval = $a['suborder'] <=> $b['suborder'];
        if ($retval == 0) {
        $retval = $a['details']['subsuborder'] <=> $b['details']['subsuborder'];
        }
    }
    return $retval;
});
1.19.27.5. uksort - custom sort by keys

1.19.28. array_reverse

$preserve_keys
if true, numeric keys are preserved. Non-numeric keys are not affected by this setting and will always be preserved
array_reverse ( array $array [, bool $preserve_keys = FALSE ] ) : array

1.19.29. array_keys, array_values

  • array_keys also has filter by array value and strict type matching (=) options
array_keys ( array $array , mixed $search_value [, bool $strict = FALSE ] ) : array
array_values ( array $array ) : array

1.19.30. array_unique 

array_unique ( array $array [, int $sort_flags = SORT_STRING ] ) : array
  • return a new array without duplicate values
  • sort_flags
    SORT_STRING
    default. the same if (string) $elem1 === (string) $elem2
    SORT_REGULAR
    don't change types. Refer to php:array_merge
    SORT_NUMERIC
    compare items numerically
    SORT_LOCALE_STRING
    compare items as strings, based on the current locale
  • Not intended to work on multi dimensional arrays
  • Keys are preserved, the first element with the same value is retained

  • Example: reverse mapping 

    $map = [
    'Annually'      => 12,
    'Semi-Annually' => 6,
    'Quarterly'     => 6, // no mapping on the receiving end
    'Monthly'       => 1,
    'Other'         => 1
];

// norway of mapping
$fromPartyA = 'Quarterly';
$toUs       = $map[ $fromPartyA ];

// reverse mapping

$reversedMap = array_flip( array_unique( $map ) );

$fromUs   = 6;
$toPartyA = $reversedMap[ $fromUs ]; // Semi-Annually

  • Sometimes continuous keys is desired for numeric array, use array_keys 

    // array_flip make sure the returned keys are unique
$unique=array_keys(array_flip(['a', 'a', 'b', 'c'])); // ['a', 'b', 'c']

1.19.31. array_flip

  • Return an array with keys eq. to the values of input array, with values eq. to the keys of the input array
    • return NULL on failure
  • The new keys need to be valid e.g. integer or string. Invalid key and its value will not be included in the result
  • The latest key will be used as its value
var_dump( array_flip( [ 'a', 'a', 'b', 'c' ] ) );
// [ a => 1, b => 2, c=> 3 ]

var_dump( array_flip( [ 'a', 'b', 'a', 'c' ] ) );
// [ a => 2, b => 1, c => 3]

1.19.32. range

// When start and end are numbers
var_export(range(2,2) === array(2));
var_export(range(2,3) === array(2, 3));
// String of numbers are converted to numbers in the result
var_export(range('2','3') === array(2, 3));

// default step is int 1
var_export(range(2,3, 1) === array(2, 3));

// step is int 3
var_export(range(2,10, 3) === array(2, 5, 8));

// can go backwards, with step being positive int 1, only based on start and end
var_export(range(2,1) === array(2, 1));

// step should be a positive number, can be float
var_export(range(2,2.2, 0.1) === array(2.0, 2.1, 2.2));

// String of numbers are converted to numbers in the result
var_export(range('2','2.2', 0.1) === array(2.0, 2.1, 2.2));

// Force the result to be string of numbers instead of numbers, although it's not perfect for floats...
// It works for strings of integers
var_export(
    array_map('strval', range('2', '2.2', 0.1)) ===
    array(
        '2', // it should be '2.0' as the original.. but oh well..
        '2.1',
        '2.2',
    )
);

// When start and end are single length characters
var_export(range('a','c') === array('a', 'b', 'c'));

var_export(range('a','c', 2) === array('a', 'c'));

// The character length of start and end should one, the following does not work though does not throw errors
var_export(range('AB','AC'));

// Use case: Generate a series of strings with prefix and number padding
var_export(
    array_map(function ($n) { return sprintf('sample_%03d', $n); }, range(1, 2)) === array('sample_001', 'sample_002')
);

1.19.33. Associative array

function isAssoc(array $arr) {
    if (array() === $arr) return false;
    return array_keys($arr) !== range(0, count($arr) - 1);
}

1.19.34. Set array value by deep path

$a = [
    'a' => [
        'b' => 123,
    ],
];

function setArrayByPath(&$array, $path, $value): array {
    $temp = &$array;
    foreach ($path as $key) {
        // if key does not exist, the value of that key is NULL
        $temp = &$temp[$key];
    }
    $temp = $value;
    return $array;
}

var_dump(($arr = setArrayByPath($a, ['a', 'b'], 'some value')) === [
        'a' => [
            'b' => 'some value',
        ],
    ]);

setArrayByPath($a, ['z', 'y', 'x'], 'some value');

var_dump($a === [
        'a' => [
            'b' => 'some value',
        ],
        'z' => [
            'y' => [
                'x' => 'some value',
            ],
        ],
    ]);

1.19.35. If it's numeric and continuous array

function hasMany($arr) {
    return array_keys($arr) === range(0, count($arr) - 1);
}

1.19.36. Custom pretty print array

function printData($array) {
    echo "<ul>";
    foreach ($array as $key => $value) {
        echo "<li>$key";
        if (is_array($value)) {
            printData($value);
        }
        else {
            echo ": " . htmlspecialchars($value);
        }
        echo "</li>";
    }
    echo "</ul>";
}
printData($data);

1.20. String

1.20.2. Empty string

// All empty values
$testCase = array(
    '',
    "",
    null,
    array(),
    FALSE,
    NULL,
    '0',
    0,
);

if (trim($str) != '') {}

1.20.3. Multibyte - refer to php:multibyte

1.20.4. String to HTML,

1.20.4.1. Parse
$dom = new DOMDocument;
$dom->loadHTML($xml);
$imgs = $dom->getElementsByTagName('img');
foreach ( $imgs as $img ) {
  echo $img->nodeValue, PHP_EOL;
  echo $img->getAttribute('src');
}
// To get the first img
$_first_img = $imgs->item(0);

echo $_first_img->getAttribute('src');
1.20.4.2. Strip one specific HTML tag
$dom = new DOMDocument;
$dom->loadHTML($htmlString);
$xPath = new DOMXPath($dom);
$nodes = $xPath->query('//*[@id="anotherDiv"]');
if($nodes->item(0)) {
    $nodes->item(0)->parentNode->removeChild($nodes->item(0));
}

$list = $doc->getElementsByTagName("p");
while ($list->length > 0) {
    $p = $list->item(0);
    $p->parentNode->removeChild($p);
}

echo $dom->saveHTML();
1.20.4.3. Add element by adding fragment

The above will add <!doctype><html><body> tags, use this to do fragment 

$dom = new DOMDocument;
$fragment = $dom->createDocumentFragment();
$fragment->appendXML($form);
$dom->appendChild($fragment);
$noscripts = $dom->getElementsByTagName('noscript');
while ($noscripts->length > 0) {
  $i = $noscripts->item(0);
  $i->parentNode->removeChild($i);
}
return $dom->saveHTML();
1.20.4.4. Output
$doc = new DOMDocument();

// set before loading elements
$doc->preserveWhiteSpace = false;
$doc->formatOutput = true;

$doc->loadHTML($r);
$doc->saveHTMLFile($filePath);

1.20.5. Regex

1.20.5.1. See 8.6.9.2.1
1.20.5.2. preg_replace
1.20.5.3. preg_match 
preg_match ( string $pattern , string $subject [, array &$matches [, int $flags = 0 [, int $offset = 0 ]]] ) : int
  • Return 1 if the pattern matches given subject, 0 if it does not, or FALSE if an error occurred
  • $matches
    $matches[0]
    the text matched the full pattern
    $matches[1]
    the text matched the first captured parenthesized subpattern and so on
1.20.5.4. preg_match_all 
preg_match_all ( string $pattern , string $subject [, array &$matches [, int $flags = PREG_PATTERN_ORDER [, int $offset = 0 ]]] ) : int
  • Return the number of full pattern matches (could be 0). Or false if an error occured
preg_match_all("|<[^>]+>(.*)</[^>]+>|U",
    "<b>example: </b><div align=left>this is a test</div>",
    $out, PREG_PATTERN_ORDER);

// $out[0] is an array of the matched results of the WHOLE pattern

echo $out[0][0] . ", " . $out[0][1] . "\n";

// <b>example: </b>, <div align=left>this is a test</div>


// $out[1] is an array of the matched results of the first parenthesized pattern

echo $out[1][0] . ", " . $out[1][1] . "\n";

// example: , this is a test
1.20.5.5. Split a multilined string by blank line
$array = preg_split("#\n\s*\n#Uis", $text);
1.20.5.6. Remove HTML tag
Refer to remove spaces between HTML start and end tags
email:spaces between html tags
// Remove HTML tags
$tags = ['noscript', 'p'];
return preg_replace('@<('. implode('|', $tags) .')[^>]*?>.*?</\\1>@si', '',$html);

// Remove an attribute from string
// e.g. remove style attribute
$output = preg_replace('/(<[^>]+) style=".*?"/si', '$1', $input);
1.20.5.7. Separate Street Name from Civic Number in Address String

https://www.htmlgoodies.com/beyond/javascript/parsing-building-and-street-fields-from-an-address-using-regular-expressions.html 

$addresses = [
    '100 Baker Street', // => '100 '
    '109 - 111 Wharfside Street', // => '109 - 111 '
    '40-42 Parkway', // => '40-42 '
    '25b-26 Sun Street', // '25b-26 '
    '43a Garden Walk', // '43'
    '6/7 Marine Road', // '6/7 '
    '10 - 12 Acacia Ave', // '10 - 12 '
    '4513 3RD STREET CIRCLE WEST', // '4513 3'
    '0 1/2 Fifth Avenue', // '0 1/2 '
    '194-03 1/2 50th Avenue', // '194-03 1/2 '
];

$re = '/^\d+\w*\s*(?:[\-\/]?\s*)?\d*\s*\d+\/?\s*\d*\s*/';

foreach ( $addresses as $k => $v ) {
    $matches = [];
    preg_match( $re, $v, $matches );
    var_dump( $matches );
}

1.20.6. Read string line by line

foreach (preg_split("/((\r?\n)|(\r\n?))/", $map) as $_line) {
    $line = explode(',', $_line); // comma separated
    foreach (explode('|', $line[1]) as $prov) {
        $r[$prov] = $line[0];
    }
}

1.20.7. String Ellipsis, Trim php:ellipsis

1.20.8. Encode HTML

  • d7:functions:check_plain is htmlspecialchars($text, ENT_QUOTES, 'UTF-8');
    • escape both single and double quotes
    • sanitize HTML attribute values (pure text)

      • JavaScript inside an HTML attribute 

        // Title should be plain text
$title = "<span style=\"color: red;\">It's supposed to be, `well`, \"plain\" text</span>";
$flags = ENT_QUOTES;
$titleEncoded = htmlspecialchars($title, $flags);

// Tips: Which means we can stack usages of htmlspecialchars()!

// A double quote "
// - htmlspecialchars first time => &quot;
// - htmlspecialchars second time => &amp;quot;
// - htmlspecialchars third time => &amp;amp;quot;

$html = <<<HTML
<p>$titleEncoded:</p>
<code>{"key": [{"fieldA": "#1"}, {"fieldB": "#2"}]}</code>
HTML;
$htmlJsonEscaped = htmlspecialchars(json_encode($html), $flags);
$jsonEscapedInScript = json_encode($html, JSON_UNESCAPED_UNICODE);
$content4 = htmlspecialchars(json_encode("I'm good, too"), $flags);
$link = '<a href="#">Click here</a>';
$data = [
    'abc' => 123,
    'a' => [
        "I'm Li",
        $link,
        // no need to escape per value
        htmlspecialchars($link),
    ],
];
$dataJsonHtmlString = htmlspecialchars(json_encode(json_encode($data)));
$dataJson = htmlspecialchars(json_encode($data));
// Use JavaScript literal for multi-line string

$fn = function ($d = '') {
    return $d;
};

$jsFunc = <<<JS
console.log({$fn(json_encode($html))}, {$fn(json_encode($data))});
JS;

var_dump(<<<HTML
<span onmouseover="{$fn(htmlspecialchars($jsFunc))}">Hover me!</span>
HTML
);

echo <<<HTML
    <script src="https://cdn.jsdelivr.net/npm/lodash@4.17.21/lodash.min.js"></script>
    <script>
        function htmlDecode(input) {
            var doc = new DOMParser().parseFromString(input, "text/html");
            return doc.documentElement.textContent;
        }

        function myFunc(content, content2, content3, content4, jsonDataString, jsonData) {
            console.log('content', content);
            document.getElementById('content').innerHTML = content;
            console.log('content2', content2);
            console.log(content3);
            console.log(content4);
            console.log(htmlDecode(content));
            console.log($jsonEscapedInScript);
            console.log(_.isEqual(JSON.parse(jsonDataString), jsonData));
            var link = '<a href="#">Click here</a>';
            console.log(jsonData.a[1] === link);
            console.log(jsonData.a[2]);
            console.log(htmlDecode(jsonData.a[2]) === link);
        }
    </script>
    <span onmouseover="myFunc($htmlJsonEscaped, 123, 'hello I\'m good', $content4, $dataJsonHtmlString, $dataJson);">Hover me!</span>
    <div id="content"></div>
HTML;

    • sanitize HTML attribute with JavaScript 

      $fileNameDisplay = htmlspecialchars($fileNameDisplay);
$fileNameDisplayAttribute = addslashes($fileNameDisplay);
$items[] = <<<HTML
<div class="listitem">
<a onclick="picklookup('{$fileNameDisplayAttribute}','{$fileId}');">{$fileNameDisplay}</a>
</div>
HTML;
    • sanitize $_POST value to safe html text for display
  • What it does?
    • function htmlspecialchars(string $string, int $flags = ENT_QUOTES|ENT_SUBSTITUTE, ?string $encoding = null, bool $double_encode = true)
      • PHP default
    • &amp;
    • &quot;, unless ENT_NOQUOTES is set
    • &#039; (for ENT_HTML401) or &apos; (for ENT_XML1, ENT_XHTML or ENT_HTML5), but only when ENT_QUOTES is set
    • &lt;
    • &gt;

  • htmlspecialchars produces less code than htmlentities 

    // Drupal's check_plain
if (!function_exists("check_plain")) {
  function check_plain($text) {
return htmlspecialchars($text, ENT_QUOTES, 'UTF-8');
  }
}

echo htmlentities('<Il était une fois un être>.');
// Output: &lt;Il &eacute;tait une fois un &ecirc;tre&gt;.
//                ^^^^^^^^                 ^^^^^^^

echo htmlspecialchars('<Il était une fois un être>.');
// Output: &lt;Il était une fois un être&gt;.
//                ^                 ^
    • htmlentities is identical to htmlspecialchars(). It also converts characters which have html:character entity equivalents
  • Flags
    ENT_QUOTES
    convert both double and single quotes
    htmlspecialchars($text, ENT_QUOTES | ENT_XML1, 'UTF-8')
    escape values for XML attributes and element data (text node)
    ENT_SUBSTITUTE
    if the text contains an invalid character, those characters are replaced with � characters, instead of returning a completely empty string.
    (no term)
    Default flags
    ENT_COMPAT | ENT_HTML401
    default flags before PHP 8.1
    ENT_COMPAT
    convert double quotes only and leave single-quotes alone
    Default from PHP 8.1
    ENT_QUOTES | ENT_SUBSTITUTE | ENT_HTML401
    • which is the same as ENT_QUOTES | ENT_SUBSTITUTE
    • Single quotes are also escaped

1.20.9. URL Encode php:urlencode

1.20.10. strip_tags

1.20.11. UTF-8 encoding 

mb_convert_encoding(array|string $string, string $to_encoding, array|string|null $from_encoding = null): array|string|false

// Auto detect source encoding and specify target encoding
$twitterHTML = mb_convert_encoding($twitterHTML, 'UTF-8');


$twitterHTML = mb_convert_encoding($twitterHTML, 'UTF-8',
  mb_detect_encoding($twitterHTML)
);
echo $twitterHTML;

1.20.12. JSON

  • Input variable is
    String
    result is wrapped with double quotes
    • with the following characters escaped by \
      • "

      • e.g. </script> becomes <\/script>

        • Perfect for assigning a PHP string variable to JavaScript variable
        <?php
$html = <<<HTML
<p></p>
HTML;
$html = json_encode($html);
?>
<script>
var a = <?php echo $html; ?>;
var html = [];
html.push(<?php echo $html; ?>);
</script>
    Number
    result is not wrapped with double quotes
    Array

    result is not wrapped with double quotes. The keys are double quoted with no escape 

    <?php
$aNumericArray = json_encode([
1,
"Hello I'm Li. \"",
'Hello I\'m Li. "'
]);
var_dump($aNumericArray);
// string with no double quotes wrapped
// [1,"Hello I'm Li. \"","Hello I'm Li. \""]

$aNumericArrayWithForceObject = json_encode([
1,
"Hello I'm Li. \"",
'Hello I\'m Li. "'
], JSON_FORCE_OBJECT);

$aNumericArrayWithPrettyPrint = json_encode([
1,
"Hello I'm Li. \"",
'Hello I\'m Li. "'
], JSON_PRETTY_PRINT);

var_dump($aNumericArrayWithForceObject);
// string with no double quotes wrapped
// {"0":1,"1":"Hello I'm Li. \"","2":"Hello I'm Li. \""}

var_dump($aNumericArrayWithPrettyPrint, '$aNumericArrayWithPrettyPrint');
// string with no double quotes wrapped
// same as $aNumericArray but with indentation

$anAssociativeArray = json_encode([
'a' => 123,
'b' => "Hello I'm Li. \"",
'c' => 'Hello I\'m Li. "',
'd' => ['a', 'b'],
]);
var_dump($anAssociativeArray);
// string with no quotes wrapped
// {"a":123,"b":"Hello I'm Li. \"","c":"Hello I'm Li. \"","d":["a","b"]}
?>

<script>
var anArray = <?php echo $aNumericArray; ?>; // a JavaScript array variable
console.log(anObj1);

var anObj = <?php echo $aNumericArrayWithForceObject; ?>; // a JavaScript object variable
console.log(anObj);

anObj = <?php echo $anAssociativeArray; ?>; // a JavaScript object variable
console.log(anObj);
</script>
    (no term)
    Object
    • By default, the following properties are included
      • Explicitly public declared properties
      • Dynamically assigned properties
  • See js:innerHTML
1.20.12.1. UTF-8
$string = 'Svrček';
var_dump($test = mb_detect_encoding($string), $test === 'UTF-8');
// detect if it's UTF-8 using preg_match
var_dump($test = preg_match('//u', $string), $test === 1);
var_dump($test = json_encode($string), $test === '"Svr\u010dek"');
var_dump($test = json_decode(json_encode($string)), $test === $string);
var_dump($test = json_encode($string, JSON_UNESCAPED_UNICODE), $test === '"' . $string . '"');
var_dump($test = json_decode(json_encode($string, JSON_UNESCAPED_UNICODE)), $test === $string);
var_dump($test = urlencode(json_encode($string, JSON_UNESCAPED_UNICODE)), $test === '%22Svr%C4%8Dek%22');
var_dump($test = urldecode(urlencode(json_encode($string, JSON_UNESCAPED_UNICODE))), $test === '"' . $string . '"');
var_dump($test = json_decode(urldecode(urlencode(json_encode($string, JSON_UNESCAPED_UNICODE)))), $test === $string);
1.20.12.2. Debug

Use json_last_error(): int and json_last_error_msg(): string to debug

  1. JSON_ERROR_UTF8 - Malformed UTF-8 characters, possibly incorrectly encoded
    • Make sure there's no binary data or non UTF-8 characters in any keys or values in json_encode($array)
1.20.12.3. If a string is a JSON string
<?php
$string = 'abc';
$number = 123;
$numberAsString = '123';
$zero = 0;
$zeroAsString = '0';
$zeroAsFloat = 0.00;
$zeroAsFloatInString = '0.00';
$null = null;
$array = array('ABC', 'xyz');

function isJSONString($string) {
    return (1 || @json_decode($string, 1)) && json_last_error() === JSON_ERROR_NONE;
}

var_dump(JSON_ERROR_NONE === 0);

var_dump(json_decode($string, 1) === null);
var_dump(json_last_error() !== JSON_ERROR_NONE);
var_dump(isJSONString($string) === false);

var_dump(json_decode($number, 1) === $number);
var_dump(json_last_error() === JSON_ERROR_NONE);
var_dump(isJSONString($number) === true);

var_dump(json_decode($numberAsString, 1) === $number);
var_dump(json_last_error() === JSON_ERROR_NONE);
var_dump(isJSONString($numberAsString) === true);

var_dump(json_decode($zero, 1) === $zero);
var_dump(json_last_error() === JSON_ERROR_NONE);
var_dump(isJSONString($zero) === true);

var_dump(json_decode($zeroAsString, 1) === $zero);
var_dump(json_last_error() === JSON_ERROR_NONE);
var_dump(isJSONString($zeroAsString) === true);

var_dump(json_decode($zeroAsFloat, 1) === $zero);
var_dump(json_last_error() === JSON_ERROR_NONE);
var_dump(isJSONString($zeroAsString) === true);

var_dump(json_decode($zeroAsFloatInString, 1) === $zeroAsFloat);
var_dump(json_last_error() === JSON_ERROR_NONE);
var_dump(isJSONString($zeroAsFloatInString) === true);

var_dump(json_decode($null, 1) === $null);
var_dump(json_last_error() !== JSON_ERROR_NONE);
var_dump(isJSONString($null) === false);

var_dump(json_decode(json_encode($array), 1) === $array);
var_dump(json_last_error() === JSON_ERROR_NONE);
var_dump(isJSONString($null) === true);
1.20.12.4. JSON Schema
  1. Keyword: type
    Data Model
    https://json-schema.org/draft/2020-12/json-schema-core.html
    (no term)
    null
    boolean
    object
    (no term)

    array 

    {
    "my_property:": {
        "type": "array",
        "items": {"type": "string"}
    }
}
    number
    base-10 decimal number. The following types are accepted but there's no distinction
    • integer
    string
    (no term)
    "type": "string" or "type": ["string", "null"]
  2. Keyword: enum
    {
    "type": "integer",
    "enum": [1, 2, 3]
}

    {
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "product": {
      "type": [
        "boolean",
        "string"
      ],
      "enum": [
        "off",
        "on",
        "semi-on",
        true,
        false
      ]
    }
  }
}
  3. Keyword: if then else
    {
  "type": "object",
  "properties": {
    "street_address": {
      "type": "string"
    },
    "country": {
      "default": "United States of America",
      "enum": ["United States of America", "Canada"]
    }
  },
  "if": {
    "properties": { "country": { "const": "United States of America" } }
  },
  "then": {
    "properties": { "postal_code": { "pattern": "[0-9]{5}(-[0-9]{4})?" } }
  },
  "else": {
    "properties": { "postal_code": { "pattern": "[A-Z][0-9][A-Z] [0-9][A-Z][0-9]" } }
  }
}
  4. Keyword: allOf
    {
  "allOf": [
    {
      "if": { ... },
      "then": { ... }
    },
    {
      "if": { ... },
      "then": { ... }
    }
  ]
}

1.20.13. Back slashes for quotes and other characters

  • Add backslashes before the following characters
    • Single quote
    • double quote
    • backslash
    • NUL (the NUL byte)
$str = "Is your name O'Reilly?";

var_dump(addslashes($str) === "Is your name O\\'Reilly?"); // Is your name O\'Reilly?

var_dump(stripslashes($str) === $str);

var_dump(stripslashes(addslashes($str)) === $str);

1.20.14. Replace php:str_replace

  • Works with multibyte
  • Search and replace all from left to right
  • use str_ireplace() for case-insensitive
  • Binary safe
  • Parameters
    $count
    it's not for limit but instead the count of replacements
str_replace( mixed $search, mixed $replace, mixed $subject [, int &$count ] ) : mixed

// single search and replace
str_replace( 'a', 'b', 'abc');

// multiple search and replace :: change a to b, x to y
str_replace( ['a','x'], ['b', 'y'], 'abcxyz');

// when $subject is an array, search and replace is performed with every entry of subject and return value is an array
var_export(str_ireplace(['n/a'], '', "N/A hello world n/a")
    === ' hello world ');
var_export(str_ireplace(['n/a'], '', ["N/A hello world n/a", "Hello World n/a N/A"])
    === [" hello world ", "Hello World  "]);

1.20.15. Replace first or last occurance 

// substr_replace ($string, $replacement, $start, $length = null) : string|string[]

function str_flreplace($search, $replace, $subject, $isFirstReplace = 1) {
    $pos = strpos($subject, $search);
    if ( ! $isFirstReplace) {
        // strrpos :: not strpos! Find the position of the last occurrence of a substring in a string
        $pos = strrpos($subject, $search);
    }

    if ($pos !== false) {
        $subject = substr_replace($subject, $replace, $pos, strlen($search));
    }

    return $subject;
}

function str_freplace($search, $replace, $subject) {
    $search = '/' . preg_quote($search, '/') . '/';

    return preg_replace($search, $replace, $subject, 1);
}

$search = '?';
$replace = 123;
$subject = "SELECT * FROM loans WHERE loanid = ? AND company = ?";

$a = str_flreplace($search, $replace ,$subject);
var_dump('$a', $a);
// "SELECT * FROM loans WHERE loanid = 123 AND company = ?"

$b = str_flreplace($search, $replace, $subject, false);
// "SELECT * FROM loans WHERE loanid = ? AND company = 123"
var_dump('$b', $b);

$c = str_freplace($search, $replace, $subject);
// "SELECT * FROM loans WHERE loanid = ? AND company = 123"
var_dump('$c', $c);

1.20.16. Find the position of the first/last occurrence of a substring in a string

Last occurance
strrpos(string $haystack, string $needle, int $offset = 0): int|false
case-insensitve
strripos(string $haystack, string $needle, int $offset = 0): int|false
First occurance
strpos(string $haystack, string $needle, int $offset = 0): int|false
case-insensitive
stripos(string $haystack, string $needle, int $offset = 0): int|false

1.20.17. First character uppercase/lowercase - ucfirst() lcfirst()

1.20.18. First letter of each word uppercase ucwords()

ucwords(string $string, string $separators = " \t\r\n\f\v"): string

1.20.19. Start with / End with

$prefix = '/inventory/';
// case-insensitive compare
if( strcasecmp(substr($request_uri, 0, strlen($prefix)), $prefix) === 0 ){
  // it has that prefix and remove the prefix
  // $new_uri = substr($request_uri, strlen($prefix));
}

// case-sensitive compare
if ( substr($request_uri, 0, strlen($prefix)) === $prefix ) {}


function endsWith($haystack, $needle) {
    $length = strlen($needle);

    return $length > 0 ? substr($haystack, -$length) === $needle : true;
}

// one liner end with
$haystack = 'path/to/file';
$needle = 'to/file';

if (strlen($needle) > 0 ? substr($haystack, -strlen($needle)) === $needle : true) {}

1.20.20. sprintf and printf 

sprintf(string $format, mixed ...$values): string

// Syntax %[flags][width][.precision]specifier

// escape % with %%

$num = 5;
$location = 'tree';

$format = 'There are %d monkeys in the %s';
echo sprintf($format, $num, $location);

// `n$` is a specifier and when it exists, it must be immediately after %

$format = 'The %2$s contains %1$d monkeys'; // The tree contains 5 monkeys

$format = 'The %2$s contains %1$04d monkeys'; // The tree contains 0005 monkeys

// `'.` is a flag, it pads the result with the character `'(char)`

echo sprintf("%'.9d\n", 123); // ......123

// I don't know.. It's the same as `echo sprintf("%09d\n", 123);`
echo sprintf("%'.09d\n", 123); // 000000123

// decimal points
echo sprintf("%.5f", 123); // 123.00000

1.20.21. strtr 

// byte-to-byte
strtr(string $string, string $from, string $to): string

// The longest keys will be tried first. Once a substring has been replaced, its new value will not be searched again 
strtr(string $string, array $replace_pairs): string

$trans = array("h" => "-", "hello" => "hi", "hi" => "hello");

// `h` is not picked because there're longer matches and replaced text was not searched again 
strtr("hi all, I said hello", $trans) === 'hello all, I said hi';

$strParams = [
    ':name' => 'Dave123',
    'Dave123' => ':name2 or :password', // Not replaced because it's not found in the original source text
    ':name2AndPassword' => ':name2 or :password',
    ':name2' => 'Steve',
    ':pass' => '7hf2348',
];

var_dump(strtr("My name is :name, not :name2. :name2AndPassword", $strParams) === "My name is Dave123, not Steve. :name2 or :password");

1.20.22. Random base64 string

$str = substr(base64_encode(sha1(mt_rand())), 0, 16);

1.20.23. Hash: md5, sha1

  • Hashed string (checksum) is a one-way transformation on the original string. Another string has a different checksum
1.20.23.1. hash()

  • Syntax 

    hash(
    string $algo,
    string $data,
    bool $binary = false,
    array $options = []
): string
  • print_r(hash_algos());
    md5
    see 1.20.23.3
    sha1
    see 1.20.23.2
    sha256
    64 lowercase hexadecimal characters (256 bits or 32 bytes) output for maximum message size of 264-1 bits
    Digest size
    the output size of the hash result
    (no term)
    2 hexdecimal characters is 1 byte
sha1($text) === hash('sha1', $text)
1.20.23.2. sha1()
  • Unsuitable for storing passwords

  • Result is a 40 lowercase hexadecimal characters (160 bits or 20 bytes). $binary = true returns binary of length 20. Maximum message size is 264-1 bits 

    sha1(string $string, bool $binary = false): string
  • numbers 0-9 and a-f (base 16 = 2^4). 1 hexadeciaml digit is 4 bit.
    9
    decimal 9
    a
    decimal 10
    b
    decimal 11
    c
    decimal 12
    d
    decimal 13
    e
    decimal 14
    f
    decimal 15
    11
    decimal 16
1.20.23.3. md5
  • Unsuitable for storing passwords
  • Output is fixed-length 128 bits = 16 bytes = 32 hexadecimal digits/numbers
<?php
// Signed cookie example

function sign_string($string)
{
    // Using $salt makes it hard to guess how $checksum is generated
    // Caution: changing salt will invalidate all signed strings
    $salt = "Simple salt";
    $checksum = sha1($string . $salt); // Any hash algorithm would work
    // return the string with the checksum at the end
    return $string . '--' . $checksum;
}

function signed_string_is_valid($signed_string)
{
    $array = explode('--', $signed_string);

    if (count($array) != 2) {
    // string is malformed or not signed
    return false;
    }

    // Sign the string portion again. Should create same
    // checksum and therefore the same signed string.
    $new_signed_string = sign_string($array[0]);
    if ($new_signed_string == $signed_string) {
    return true;
    } else {
    return false;
    }
}

// Uncomment to demonstrate usage

$string = "This is a test.";
echo "Original: " . $string . "<br />";
echo "<br />";

$signed_string = sign_string($string);
echo "Signed: " . $signed_string . "<br />";
echo "<br />";

$valid = signed_string_is_valid($signed_string);
echo "Valid? " . ($valid ? 'true' : 'false') . "<br />";
echo "<br />";

$tampered_string = "This was a test.--521d61df5d4c239a1f5a191ff3803826b60587a9";
echo "Tampered: " . $tampered_string . "<br />";
echo "<br />";

$valid = signed_string_is_valid($tampered_string);
echo "Valid? " . ($valid ? 'true' : 'false') . "<br />";
echo "<br />";
1.20.23.4. hash_hmac 
hash_hmac(
    string $algo,
    string $data,
    string $key,
    bool $binary = false
): string
  • Generate a keyed hash value using HMAC method
  • MAC (Message Authentication Code) is produced from a message and a secret key by a MAC algo.
    • It's impossible to produce the MAC of a message and a secret key without knowing the secret key
    • A MAC of the same message produced by a different key looks unrelated
    • Even knowing the MAC of other messages does not help in computing the MAC of a new message
    • Example, our server receives a long message and a MAC, the server uses the message and the secret key to compute the MAC and finds it's the same as the provided MAC, then the server sees this message comes from a valid source. A MAC is also called a signature based on secret key.
  • HMAC is a MAC based on a hash function
    • There're other ways to define a MAC e.g. MAC algo based on block ciphers such as CMAC
hash_hmac('md5', 'The quick brown fox jumps over the lazy dog', 'key'); // 32 hexadecimal hash

1.20.24. Password Hash 

if ( isset( $_POST['submit'] ) ) {
    $password = $_POST['password'];
    echo "Text password: " . $password . "<br />";

    // Encryption example
    $hashed_password = password_hash( $password, PASSWORD_BCRYPT );
    // default PASSWORD_DEFAULT now actually uses PASSWORD_BCRYPT, should stick with default as
    // PHP will upgrade the underlying algo in the future
    echo "Hashed password: " . $hashed_password . "<br />";
    echo "<br />";

    // Verification example
    $wrong_password = "anything_else";
    $is_match       = password_verify( $wrong_password, $hashed_password );
    echo "Attempt 1: " . ( $is_match ? 'correct' : 'wrong' ) . "<br />";

    $is_match = password_verify( $password, $hashed_password );
    echo "Attempt 2: " . ( $is_match ? 'correct' : 'wrong' ) . "<br />";
    echo "<br />";
    echo "<hr />";

    // rehash
    $opt = ['cost' => 12 ];
    if (password_needs_rehash($password, PASSWORD_DEFAULT, $opt)) {
        $newHash = password_hash($password, PASSWORD_DEFAULT, $opt);
    }
}

1.20.25. encrypt, decrypt

  • should be used instead of

  • Encrypt 

    $data = "hello";
$method = "aes-256-cbc";
$key = "123455";
$iv = "1234567890123456"; // has to be 16 bytes or 16 text characters
$iv = openssl_random_pseudo_bytes(16); // $iv should be different every time

$options = 0; // If $data, $key and $iv are normal text like above, 0 can be used. Output is base64 encoded string
// Otherwise use OPENSSL_RAW_DATA. Output is binary

$encryptedMessage = openssl_encrypt($data, $method, $key, $options, $iv);

echo $encryptedMessage.PHP_EOL; // because $options is 0, the encrypted message is base64 encoded

$decryptedMessage = openssl_decrypt($encryptedMessage, $method, $key, $options, $iv);

  • Decrypt 

    $fromNode = 'encryptedString$intializedVector';
$fromNode = explode('$', $fromNode);
$key = '64 hex characters'; // same as in nodejs

$data = hex2bin($fromNode[0]);
$keyAES = hex2bin($key);
$iv = hex2bin($fromNode[1]);
$options = OPENSSL_RAW_DATA; // Input are all binary, output is binary, too. 
// But because the encrypted message is a string, the decrypted message is a string.

$decrypted = openssl_decrypt($data, 'aes-256-cbc', $keyAES, $options, $iv);

// If $data, $keyAES and $iv are raw data (bytes), use OPENSSL_RAW_DATA as $options
// if all 3 of them are string, use 0 as $options
    • If $encryptedMessage, $key and $iv are all hex, you need to convert them to binary to decrypt
    • Refer to nodejs:decrypt

1.20.26. Crypt 

crypt(string $string, string $salt): string
  • It creates a one-way hashed string and it is not yet binary safe
  • $salt
    • Based on the salt provided, it will create a hash using the corresponding algorithm
      • Can be omitted but not recommended. When omitted, PHP will auto-generate a 12 character salt using MD5 algo if it's available, if not, a 2 character salt using DES algo
    • 2 character salt from the alphabet ./0-9A-Za-z
    • 9 character salt starting with _ followed by 4-character-iteration-count and 4-character-salt
    • 12 character salt starting with $1$
    • CRYPT_BLOWFISH
      Waring !!
      using this algo, the $string is truncated to 72 bytes!
    • 16 character salt prefixed with $5$, may be followed by rounds=<N>$ where N is used to indicate how many times the hashing loop should be executed, default is 5000, with minimum 1000 and maximum of 999999999
    • same as CRYPT_SHA256 but prefix is $6$
$user_input = 'rasmuslerdorf';
$hashed_password_stored_in_db = crypt($user_input, '$5$rounds=5000$usesomesillystringforsalt$');
// Only the first 16 characters of the salt will be used: usesomesillystri
$hashed_password_stored_in_db = '$5$rounds=5000$usesomesillystri$KqJWpanXZHKq2BOB43TSaYhEWsQ1Lr5QNyPCDH/Tp.6';
// KqJWpanXZHKq2BOB43TSaYhEWsQ1Lr5QNyPCDH/Tp.6 is the hashed result
var_dump(hash_equals($hashed_password_stored_in_db, crypt($user_input, $hashed_password_stored_in_db)));

1.20.27. Random

function randomString(int $length, $options = array()): string {
    if ($options['intOnly'] ?? null) {
        return strval(random_int(pow(10, $length - 1), pow(10, $length) - 1));
    }
    $returnBase64 = function($rand) use ($length, $options) {
        $r = substr(base64_encode($rand), 0, $length);
        return ($options['lowerCase'] ?? null) ? strtolower($r) : $r;
    };
    if (is_callable('random_bytes')) {
        return $returnBase64(random_bytes($length * 2));
    }
    if (is_callable('openssl_random_pseudo_bytes')) {
        return $returnBase64(openssl_random_pseudo_bytes($length * 2));
    }
    return $returnBase64(mt_rand());
}

1.20.28. True or False on string

// index.php?
var_dump($_GET['notexist']); // null
// index.php?abc=
var_dump($_GET['abc']); // "" empty string
// So $_GET['abc'] can be a string or null
var_dump(trim(null)); // "" empty string
// trim($_GET['abc']) returns only string
$b = '';
var_dump(($b) ? true : false); // false
$a = [
    '' => false,
    '0' => false,
        '1' => true,
        '-1' => true,
];

1.20.29. If string is an integer

// Option #1.
// Limitation: leading zero string e.g. '05' is not considered an integer
if ($value === strval(intval($value))) {
    return intval($value);
}

var_dump(filter_var('2', FILTER_VALIDATE_INT));   // 2
var_dump(filter_var('2.0', FILTER_VALIDATE_INT)); // false
var_dump(filter_var('2.1', FILTER_VALIDATE_INT)); // false

var_dump(filter_var(2, FILTER_VALIDATE_INT));     // 2
var_dump(filter_var(2.0, FILTER_VALIDATE_INT));   // 2
var_dump(filter_var(2.1, FILTER_VALIDATE_INT));   // false

1.21. Conversion to boolean

/*
 * https://www.php.net/manual/en/language.types.boolean.php
 */

var_dump(0 == false);
var_dump(0.0 == false);
var_dump(-0.0 == false);
var_dump('0' == false);
var_dump('' == false);
var_dump([] == false);
var_dump(null == false);

// SimpleXML objects created from attributeless empty elements, i.e. elements which have neither children nor attributes.

1.22. URL

1.22.1. urlencode() :: encode the query string (the part after ?)

  • encodes spaces as plus signs
  • eq. to js:encodeURIComponent
  • Should be used to encode keys and values pairs of POST data
$query_string = 'foo=' . urlencode($foo) . '&bar=' . urlencode($bar);
echo '<a href="mycgi?' . htmlentities($query_string) . '">';

1.22.2. rawurlencode

  • encode path (the part before ?). Almost the same as urlencode except it enocodes space as %20

  • Basics 

    echo '<a href="ftp://user:', rawurlencode('foo @+%/'),
 '@ftp.example.com/x.txt">';
echo '<a href="http://example.com/department_list_script/',
rawurlencode('sales and marketing/Miami'), '">';

  • Use to encode filename in header:content-disposition 

    function fileNameForHeader( $fileName = '' ) {
    // Usage: header("Content-disposition: attachment; filename=\"fileNameForHeader('FileName')\"");
    $fileName = str_replace( array( "\n", "\r", "\r\n", ' ', ',' ), '_', $fileName );
    $fileName = str_replace( array( "'", '"' ), '', $fileName );

    return rawurlencode( $fileName );
}

1.22.3. http_build_query

  • Return a URL-encoded query string
    • By default perfect for cURL POST content-type application/x-www-form-urlencoded
      • spaces are encoded as + which is eq. to urlencode
      • Also good for GET url query string
var_dump(http_build_query([
    'a' => 'hello world', // a=hello+world
    'b' => ' hello world ', // &b=+hello+world+
    'c' => [1, 2, 3], // &c%5B0%5D=1&c%5B1%5D=2&c%5B2%5D=3 which is &c[0]=1&c[1]=2&c[2]=3
    'd' => ['1' => 'hello', '2' => 'world'], // &d%5B1%5D=hello&d%5B2%5D=world which is &d[1]=hello&d[2]=world
    'e' => ['1' => ' hello ', '2' => 'wor ld'], // &e%5B1%5D=+hello+&e%5B2%5D=wor+ld which is &e[1]=+hello+&e[2]=wor+ld
    'f' => ['hello' => 'world', 'hello2' => 'world'], // &f%5Bhello%5D=world&f%5Bhello2%5D=world which is &f[hello]=world&f[hello2]=world
    'g' => [' hello ' => ' world ', 'hello 2' => 'wor ld'], // &g%5B+hello+%5D=+world+&g%5Bhello+2%5D=wor+ld which is &g[+hello+]=+world+&g[hello+2]=wor+ld
    'h' => json_encode([' hello ' => ' world ', 'hello 2' => 'wor ld']), // &h=%7B%22+hello+%22%3A%22+world+%22%2C%22hello+2%22%3A%22wor+ld%22%7D which is &h={"+hello+":"+world+","hello+2":"wor+ld"}
    ' hello ' => [' hello ' => ' world ', 'hello 2' => 'wor ld'], // &+hello+%5B+hello+%5D=+world+&+hello+%5Bhello+2%5D=wor+ld which is &+hello+[+hello+]=+world+&+hello+[hello+2]=wor+ld
    'wor ld' => [' hello ' => ' world ', 'hello 2' => 'wor ld'], // &wor+ld%5B+hello+%5D=+world+&wor+ld%5Bhello+2%5D=wor+ld which is &wor+ld[+hello+]=+world+&wor+ld[hello+2]=wor+ld
]));

1.22.4. parse_str :: parse query parameter value as an array

$url = $_SERVER['REQUEST_URI'];
$parts = parse_url($url);
if ($parts !== false) {
    // if there's error, it will return false
    parse_str($parts['query'], $query);
    echo $query['email'];
}

// parse_str ($str, array &$result = null) : void
$str = "first=value&arr[]=foo+bar&arr[]=baz";
parse_str($str, $output);
echo $output['first'];  // value
echo $output['arr'][0]; // foo bar
echo $output['arr'][1]; // baz

function removeOneQueryParameter($url, $queryParameter)
{
    if (($urlParsed = parse_url($url)) === false) {
        return $url;
    }
    parse_str($urlParsed['query'], $query);
    unset($query[$queryParameter]);
    $queryPart = http_build_query($query);
    return strtok($url, '?') . ($queryPart ? "?{$queryPart}" : '');
}

var_dump(removeOneQueryParameter('https://abc.com?abc=123', 'abc') === 'https://abc.com');
var_dump(removeOneQueryParameter('https://abc.com?xyz=&abc=123', 'abc') === 'https://abc.com?xyz=');

1.22.5. Current URL path. See php:parse_url

$path_only = parse_url($_SERVER['REQUEST_URI'], PHP_URL_PATH);

1.22.6. The last component of path

basename("/etc/passwd") // passwd
basename("/etc/sudoers.d") // sudoers.d
basename("/etc/sudoers.d", ".d") // sudoers

1.22.7. Add URL Parameter

if (!function_exists('http_build_url')) {

  /**
   * URL constants as defined in the PHP Manual under "Constants usable with
   * http_build_url()".
   *
   * https://github.com/jakeasmith/http_build_url/blob/master/src/http_build_url.php
   * @see http://us2.php.net/manual/en/http.constants.php#http.constants.url
   */
  if (!defined('HTTP_URL_REPLACE')) {
    define('HTTP_URL_REPLACE', 1);
  }
  if (!defined('HTTP_URL_JOIN_PATH')) {
    define('HTTP_URL_JOIN_PATH', 2);
  }
  if (!defined('HTTP_URL_JOIN_QUERY')) {
    define('HTTP_URL_JOIN_QUERY', 4);
  }
  if (!defined('HTTP_URL_STRIP_USER')) {
    define('HTTP_URL_STRIP_USER', 8);
  }
  if (!defined('HTTP_URL_STRIP_PASS')) {
    define('HTTP_URL_STRIP_PASS', 16);
  }
  if (!defined('HTTP_URL_STRIP_AUTH')) {
    define('HTTP_URL_STRIP_AUTH', 32);
  }
  if (!defined('HTTP_URL_STRIP_PORT')) {
    define('HTTP_URL_STRIP_PORT', 64);
  }
  if (!defined('HTTP_URL_STRIP_PATH')) {
    define('HTTP_URL_STRIP_PATH', 128);
  }
  if (!defined('HTTP_URL_STRIP_QUERY')) {
    define('HTTP_URL_STRIP_QUERY', 256);
  }
  if (!defined('HTTP_URL_STRIP_FRAGMENT')) {
    define('HTTP_URL_STRIP_FRAGMENT', 512);
  }
  if (!defined('HTTP_URL_STRIP_ALL')) {
    define('HTTP_URL_STRIP_ALL', 1024);
  }

  /**
   * Build a URL.
   *
   * The parts of the second URL will be merged into the first according to
   * the flags argument.
   *
   * @param mixed $url (part(s) of) an URL in form of a string or
   *                       associative array like parse_url() returns
   * @param mixed $parts same as the first argument
   * @param int $flags a bitmask of binary or'ed HTTP_URL constants;
   *                       HTTP_URL_REPLACE is the default
   * @param array $new_url if set, it will be filled with the parts of the
   *                       composed url like parse_url() would return
   *
   * @return string
   */
  function http_build_url($url, $parts = [], $flags = HTTP_URL_REPLACE, &$new_url = []) {
    is_array($url) || $url = parse_url($url);
    is_array($parts) || $parts = parse_url($parts);

    isset($url['query']) && is_string($url['query']) || $url['query'] = NULL;
    isset($parts['query']) && is_string($parts['query']) || $parts['query'] = NULL;

    $keys = ['user', 'pass', 'port', 'path', 'query', 'fragment'];

    // HTTP_URL_STRIP_ALL and HTTP_URL_STRIP_AUTH cover several other flags.
    if ($flags & HTTP_URL_STRIP_ALL) {
  $flags |= HTTP_URL_STRIP_USER | HTTP_URL_STRIP_PASS
    | HTTP_URL_STRIP_PORT | HTTP_URL_STRIP_PATH
    | HTTP_URL_STRIP_QUERY | HTTP_URL_STRIP_FRAGMENT;
    }
    elseif ($flags & HTTP_URL_STRIP_AUTH) {
  $flags |= HTTP_URL_STRIP_USER | HTTP_URL_STRIP_PASS;
    }

    // Schema and host are alwasy replaced
    foreach (['scheme', 'host'] as $part) {
  if (isset($parts[$part])) {
    $url[$part] = $parts[$part];
  }
    }

    if ($flags & HTTP_URL_REPLACE) {
  foreach ($keys as $key) {
    if (isset($parts[$key])) {
      $url[$key] = $parts[$key];
    }
  }
    }
    else {
  if (isset($parts['path']) && ($flags & HTTP_URL_JOIN_PATH)) {
    if (isset($url['path']) && substr($parts['path'], 0, 1) !== '/') {
      // Workaround for trailing slashes
      $url['path'] .= 'a';
      $url['path'] = rtrim(
      str_replace(basename($url['path']), '', $url['path']),
      '/'
        ) . '/' . ltrim($parts['path'], '/');
    }
    else {
      $url['path'] = $parts['path'];
    }
  }

  if (isset($parts['query']) && ($flags & HTTP_URL_JOIN_QUERY)) {
    if (isset($url['query'])) {
      parse_str($url['query'], $url_query);
      parse_str($parts['query'], $parts_query);

      $url['query'] = http_build_query(
        array_replace_recursive(
      $url_query,
      $parts_query
        )
      );
    }
    else {
      $url['query'] = $parts['query'];
    }
  }
    }

    if (isset($url['path']) && $url['path'] !== '' && substr($url['path'], 0, 1) !== '/') {
  $url['path'] = '/' . $url['path'];
    }

    foreach ($keys as $key) {
  $strip = 'HTTP_URL_STRIP_' . strtoupper($key);
  if ($flags & constant($strip)) {
    unset($url[$key]);
  }
    }

    $parsed_string = '';

    if (!empty($url['scheme'])) {
  $parsed_string .= $url['scheme'] . '://';
    }

    if (!empty($url['user'])) {
  $parsed_string .= $url['user'];

  if (isset($url['pass'])) {
    $parsed_string .= ':' . $url['pass'];
  }

  $parsed_string .= '@';
    }

    if (!empty($url['host'])) {
  $parsed_string .= $url['host'];
    }

    if (!empty($url['port'])) {
  $parsed_string .= ':' . $url['port'];
    }

    if (!empty($url['path'])) {
  $parsed_string .= $url['path'];
    }

    if (!empty($url['query'])) {
  $parsed_string .= '?' . $url['query'];
    }

    if (!empty($url['fragment'])) {
  $parsed_string .= '#' . $url['fragment'];
    }

    $new_url = $url;

    return $parsed_string;
  }
}

function _lili_campaign_url( $link, $options = []) {
    if (!empty($options) && isset($options['utm_campaign'])) {
        $default = [
            'utm_source' => 'newcom', // default to newcom, $options can override $default
            'utm_medium' => 'email', // default to email, $options can override $default
            // 'utm_campaign' = 'provided', // if utm_campaign is not set, it will return the original link
            'utm_content' => date('YmdGis'), // default to current date and time, $options can override $default
        ];
        $options = array_merge( $default, $options );

        $url = @parse_url($link);
        if (!empty($url)) {
            parse_str($url['query'], $params);
            // existing url parameters will remain regardless of $default and $options
  $params = array_merge($options, $params);
  // move utm_* parameters to the end
  foreach ($options as $item => $value) {
    $v = $params[$item];
    unset($params[$item]);
    $params[$item] = $v;
  }
  // url parameter values are encoded
            $url['query'] = http_build_query($params);
            $link = http_build_url($url);
        }
    }

    return $link;
}

1.23. Random Integer Number

There're several ways 

$numbers = range(0,12); // 0, 1, 2, ..., 12
shuffle($numbers);
echo $numbers[0];

echo(mt_rand(10,100)); // min: 0, max: 100
$rnd = rand(pow(10, 8-1), pow(10, 8)-1); // random 7 digits integer

1.24. HTML encode and decode

$t = '<a href="http://expample.com">Hello L\'automobile from "Me"</a>';
var_dump(htmlentities($t, ENT_QUOTES, 'UTF-8'));
var_dump(html_entity_decode($t, ENT_QUOTES, 'UTF-8'));
// The same as D7 decode_entities($t)

ENT_QUOTES is to convert both single &#039; and double &quot; quotes.

1.25. Heredoc, Nowdoc

1.25.1. Heredoc

  • Result is a double-quoted string
  • Limitation

    • Can't interpolate constants, class constants and class static properties directly. Use sprintf, printf, or $fn instead 

      <?php

function echoSomething($text) {
    echo $text;
}

class A {

    const NAME = 'abc';
    public static $abc = 'xyz';

    public static function hello() {
        $fn = function ($d = '') {
            return $d;
        };
        $sum = 0;
        echo <<<TEXT
{$fn(self::NAME)}
{$fn(self::$abc)}

{$fn((function ($abc) {
    $r = '';
    if ($abc) {
        $r = $abc;
    }
    return $r;
})(self::$abc))}

{$fn((function ($abc) {
    ob_start();
    echoSomething($abc);
    return ob_get_clean();
})(self::$abc))}

{$fn((function ($array) {
    array_walk($array, function ($v, $k) use (&$r) {
        $r .= <<<HTML
    <span>{$k}: {$v}</span>
HTML;
    }, $r = '');
    return $r;
})(array('an', 'array', 'to walk/loop')))}

{$fn(
    /*
 Comment out with <?php blocks
        <?php
        $a  = 1
        ?>
    */
)}

{$fn((function ($results, $fn) {
    $f = function($array) use (&$f, $fn) {
        return <<<HTML
<ul>
    {$fn((function ($array, $f, $fn) {
        array_walk($array, function ($v, $k) use (&$r, $f, $fn) {
        $r .= <<<HTML
    <li>
    {$fn(htmlspecialchars($k))}
    {$fn(is_array($v) ? $f($v) : ': ' . htmlspecialchars($v))}
HTML;
        }, $r = '');
        return $r;
    })($array, $f, $fn))}
</ul>
HTML;
    };
    return $f($results);
})(['a' => ['b' => 'c']], $fn))}

{$fn("New Sum is: " . ($sum = $sum + 1) . PHP_EOL)}

{$fn("New Sum is: " . $newSum = (function ($sum) {
    return ++$sum;
})($sum))}

{$fn((list($newSum, $newSumHTML) = (function ($sum) {
    return array(++$sum, '<p>HTML</p>');
})($sum)) == '')}

{$fn("New Sum is: {$newSum}" . PHP_EOL)}

{$fn($newSumHTML)}

TEXT;
    }

}

A::hello();

  • Query 

    $params = [];

$query = <<<MYSQL
SELECT *
FROM myTable
WHERE
  aCol = {$fn(
    ($params = array_merge($params, array($a))) ? '?' : ''
)}
  bCol = {$fn(
    ($params[] = [$a]) ? '?' : ''
)}
MYSQL;
1.25.1.1. Complex (curl) syntax
$juice = 'apple';
$a = array(
    'a' => 'abc',
);
$referToVariableJuice = 'juice';
function getVariableName() {
    return 'juice';
}

function returnArray($data = null) {
    return isset($data) ? $data : array('a' => 'abc');
}

$lambda = function ($data) {
    return $data;
};
$getNameLambda = function () {
    return 'getNameLambda';
};

class Foo {
    const NAME = 'name: foo';
    public $bar = 'I am bar.';
    public static $staticProperty = 'staticProperty';
}

$foo = new Foo();
$bar = 'bar';
$baz = array('foo', 'bar', 'baz', 'quux');

echo <<<HTML
Simple:
$juice => apple, use \$juice to escape

Dollar sign simple syntax: 
${a['a']} => abc, use \${a['a']} to escape 
${juice} => apple, use \${juice} to escape
${juice}s => apples

Complex (curl) syntax:
{$a['a']} => abc, use {\$a['a']} to escape
{$juice}s => apples, 
{${$referToVariableJuice}} => ${$referToVariableJuice} => apple

Complex (curl) syntax access class properties using variables:
{$foo->bar} => I am bar.
{$foo->{$baz[1]}} => {\$foo->bar} => {$foo->bar} => I am bar.

Complex (curl) syntax can call function!
{${getVariableName()}} => \${getVariableName()} (syntax error on phpStorm and PHP 5.4-) => apple => \${juice}
{$getNameLambda()} => getNameLambda
{$lambda(time())} => Unix timestamp
{$lambda(time() / 2)} => Unix timestamp
{$lambda(time() . ' ' . $lambda( time() / 2 ))}
{$lambda(returnArray()['a'])} => abc
{$lambda(returnArray(array('a' => 'xyz'))['a'])} => xyz
{$lambda((returnArray()['a']) ? time() : time() / 2)} => Unix timestamp
{$lambda(Foo::NAME)} => name: foo
{$lambda(Foo::$staticProperty)} => staticProperty

PHP v7.0 below
{$lambda(
    call_user_func(
        function ($delimiter) {
            $r = '';
            foreach (array('hello', 'world') as $i) {
                $r .= <<<HTML
<p>{$i}</p> {$delimiter}
HTML;
            }

            return $r;
        },
        $delimiter
    )
)}

PHP v7.0+
{$lambda(
    (function ($delimiter) {
        $r = '';
        foreach (array('hello', 'world') as $i) {
            $r .= <<<HTML
<p>{$i}</p> {$delimiter}
HTML;
        }

        return $r;
    })(
        $delimiter
    )
)}
HTML;

// lambda can be nested
{$lambda(
    call_user_func(
        function ($delimiter, $lambda) {
            $r = '';
            foreach (array('hello', 'world') as $i) {
                $r .= <<<HTML
<p>{$i}</p> {$lambda(date('Y-n-j'))}
HTML;
            }

            return $r;
        },
        $delimiter, $lambda
    )
)}
1.25.1.2. Use sprintf
$html = <<<HTML
<span>{$obj->name[1]}</span>
HTML;

// this is compatible with PHP 5.3+ including 5.3
$query = sprintf(<<<MYSQL
SELECT * FROM t1 WHERE id IN (%s, %s)
MYSQL
, $a, $a);

// For PHP 7.3+ and including 7.3
$query = sprintf(<<<MYSQL
SELECT * FROM t1 WHERE id IN (%s, %s)
MYSQL, $a, $a);

$string = <<<heredoc
plain text and now a function: %s
heredoc;
$string = sprintf($string, testfunction());

1.25.2. Nowdoc can't expand variables and the result is single quoted string

echo <<<'EOT'
<span>{$obj->name[1]}</span>
EOT;

1.25.3. Alternative

  • Also provides syntax highlighting and code inspection in PhpStorm
$a = "
    <div class=\"lookupanchor\">
        <div class=\"minilookup\" id=\"{$id}_lookup\">
        <a id=\"{$id}_lookup_closer\" class=\"labelbutton closer\" onclick=\"gid('{$id}_lookup').style.display='none';\">x</a>
        <div id=\"{$id}_lookup_view\" class=\"lookupview\" $styles></div>
        </div>
    </div>";

1.25.4. Regex to find closing tag

(?:SQL[\s]+$|SQL[)},]+|^\s+SQL)

1.26. Buffer

http://php.net/manual/en/book.outcontrol.php https://phpfashion.com/everything-about-output-buffering-in-php Send headers to browser after php has begun outputting data. 

// set header
ob_start();
echo "Hello\n";

setcookie("cookiename", "cookiedata");

// the same header cannot be set because it's already been sent
ob_end_flush();

ob_start() can be stacked

Flush means send to output

ob_end_flush
send and turn off output buffering
ob_get_flush
send, return and turn off
ob_flush
send
ob_clean
erase the output buffer
ob_end_clean
erase and turn off
ob_get_clean
return, erase and turn off eq. to ob_get_contents + ob_end_clean
ob_get_contents
return

ob_get_length 

ob_start();
?>
    <div id="site_header">
    <div id="site_header_inner">
        <div id="site_header_logo"><?php echo $PhpVar; ?></div>
        <div id="site_header_countdown">BARE XX DAGER IGJEN</div>
    </div>
    </div>
<?php
$output = ob_get_clean();
//ob_flush();

return $output;

function mmm_wpml_shortcode_func(){
    ob_start();
    do_action('icl_language_selector');
    $output_string = ob_get_contents();
    ob_end_clean();

    return $output_string;
}

1.27. Template System

static public function renderHeader($data = array()) {
  $localData = array();
  return self::renderView(__DIR__ . '/../views/header.php', $localData);
}

static private function renderView($path, $data=array()) {
  ob_start();
  if (file_exists($path)) {
    include ($path);
  }
  // else throw new TemplateNotFoundException();
  return ob_get_clean();
}
header.php

1.28. Image

1.28.1. GD vs Imagick

  • GD is the oldest. Imagick might not exist in some hosting
  • GD only supports JPG, PNG, GIF, WBMP, WebP, XBM and XPM
  • If TIFF needs to be supported, use Imagick
if(extension_loaded('gd')) {
    print_r(gd_info());
}
else {
    echo 'GD is not available.';
}

if(extension_loaded('imagick')) {
    $imagick = new Imagick();
    print_r($imagick->queryFormats());
}
else {
    echo 'ImageMagick is not available.';
}

1.28.2. Create image in memory and output

$data = 'iVBORw0KGgoAAAANSUhEUgAAABwAAAASCAMAAAB/2U7WAAAABl'
   . 'BMVEUAAAD///+l2Z/dAAAASUlEQVR4XqWQUQoAIAxC2/0vXZDr'
   . 'EX4IJTRkb7lobNUStXsB0jIXIAMSsQnWlsV+wULF4Avk9fLq2r'
   . '8a5HSE35Q3eO2XP1A1wQkZSgETvDtKdQAAAABJRU5ErkJggg==';
$data = base64_decode($data);

$im = imagecreatefromstring($data);
if ($im !== false) {
  header('Content-Type: image/png');
  imagepng($im);
  imagedestroy($im);
}
else {
    echo 'An error occurred.';
}

I think header:cache-control is automatically set to no cache To set Content-Length: 

ob_start();
imagejpeg($img);

header('Content-Type: image/jpg');
header("Content-length: " . ob_get_length()); 
// images do not compression e.g. gzip

imagedestroy($im);
// send it
ob_flush();

1.28.3. Resize image

SimpleImage - GD library 

require 'SimpleImage.php';
try {
  // Create a new SimpleImage object
  $image = new \claviska\SimpleImage();
  //$new_file = __DIR__ . '/'.$file_path.$file_no_ext.'_200x200f.'.$ext;
  $new_file = $file_path.$file_no_ext.'_200x200f.'.$ext;

  $image
    ->fromFile($file_orig_full)              // load parrot.jpg
    //->autoOrient()                        // adjust orientation based on exif data
    ->bestFit(200, 200)
    ->toFile($new_file);                         // output to the screen
  $localW = $image->getWidth();
  $localH = $image->getHeight();
  var_dump($localW, $localH);
} catch(Exception $err) {
  // Handle errors
  echo $err->getMessage();
}

1.29. Apply XSLT on XML

$xml = new DOMDocument;
$xml->load('cdcatalog.xml');

$xsl = new DOMDocument;
$xsl->load('cdcatalog.xsl');

$proc = new XSLTProcessor;
$proc->importStyleSheet($xsl);

echo $proc->transformToXML($xml);

1.30. Block Referral Traffic

if ( isset( $_SERVER['HTTP_REFERER'] ) ) {
  $_referer_domain = array(
    "~timebie\.com~i",
    "~ana-white\.com~i",
    "~mlizcochico\.com~i",
    "~adishofdailylife\.com~i",
    "~techvibes\.com~i",
    "~sociableblog\.com~i",
    "~laurenconrad\.com~i",
    "~gingerhotels\.com~i",
    "~grammarist\.com~i",
    "~dobbersports\.com~i",
    "~netsidebar\.com~i",
    "~myhomeideas\.com~i",
    "~fhm\.com~i",
    "~gamezebo\.com~i"
  );

  foreach ( $_referer_domain as $_referer ) {
    if ( preg_match( $_referer, $_SERVER['HTTP_REFERER'] ) ) {
  exit;
    }
  }
  // php:parse_url
  $_referrer = parse_url($_SERVER['HTTP_REFERER']);
  if ($_referrer !== false && !is_null($_referrer['host'])
  && preg_match( "~oralhealthgroup\.com~i", $_referrer['host'] )) {
    $_p = (isset($_GET['p'])) ? $_GET['p'] : '';
    $_subid = (isset($_GET['subid'])) ? $_GET['subid'] : '';
    $_uid = (isset($_GET['uid'])) ? $_GET['uid'] : '';
    if ($_p !== '' && $_subid !== '' && $_uid !== '') {
  exit;
    }
  }
}

1.31. PHPDoc

1.31.2. DocBlock, Structural Elements

  • The docblock has to be immediately above any Structural Element which include
    • file
    • require(_once)
    • include(_once)
    • class
    • interface
    • trait
    • function (including methods)
    • property
    • constant
/**
 * One-line summary, ending in a period (.).
 *
 * Additional paragraph of explanation.
 * Additional paragraph of explanation.
 *
 * @since 3.1.0
 *
 * @param string $mail
 *   The email address. The description can be longer if necessary, and if so,
 *   you can wrap it to another line. Indented by 2 spaces
 * @param string $from
 *   (optional) The email address to send the mail from, if different from
 *   the site-wide address.
 * @param string $bcc One line description.
 *
 * @return int One line description.
 */

1.31.3. Order of sections

  • One-line summary ending in a period .
  • Addtional paragraphs of explanation
  • PHPDoc:@var
  • only for function
  • empty line
  • @return
  • @throws
  • @ingroup
  • @deprecated
  • PHPDoc:@see
  • @todo
  • @Plugin
  • not included in standard, I use this to document side effects

Separate different-type sections by a blank line 

// blank line
@param string $param1
@param int $param2
// blank line
@return

1.31.4. Tag reference

1.31.4.1. @name
/**
 * Now, when @global is used in a function, it will link to $baz
 * @name $baz
 * @global array $GLOBALS ['baz']
 */
$GLOBALS['baz'] = array('foo', 'bar');

/**
 * @global array used for stuff
 */
function mine() {
    global $baz;
}
1.31.4.2. @code
* Example usage:
* @code
* mymodule_print('Hello World!');
* @endcode
* Text to immediately follow the code block.
1.31.4.3. @var
  • PHPDoc:datatype

  • variable name is not required 

    class FooBar {

  /**
   * The database connection object for this statement's DatabaseConnection.
   *
   * @var \Drupal\Core\Database\Connection
   */
  public $dbh;

}

  • variable name is required! 

    /** @var \Sqlite3 $sqlite */
foreach ($connections as $sqlite) {
    // there should be no docblock here
    $sqlite->open('...');
}

  • variable name is required! 

    // e.g. in `include` and `include_once`
// or an alias is desired

// variable name is required

/** @var \Sqlite3 $sqLite */
$sqlLite = $this;

/** @var int description goes here, no variable name */
$intMyVar = 1;
1.31.4.4. @link 
// @link URL <link text>
// @link URL1, URL2, URL3...

// Use this to show mutation. Not standard but I find it convenient

class ABC {
    /**
     * @var string
     */
    private string $firstVar;

    /**
     * Calls parent constructor, then increments {@link $firstVar}
     *
     * or
     *
     * @link $firstVar, $secondVar
     *
     * @echo string
     */
    public function a() {
        $this->firstVar = '';
        echo 'das';
    }

}
1.31.4.5. @see PHPDoc:@see
@see [URI | FQSEN] [<description>]
// or inline
{@see [URI | FQSEN] [<description>]}

@see AnotherClassName::Methodname()
//or
@see AnotherClassName::$varname
@see elementname() // to search a function or method
@see $elementname // to search var in the current class

// Inline tag:

/**
 * @return int Indicates the number of {@see \Vendor\Package\ClassName}
 *             items.
 */
function count() {
    <...>
}
1.31.4.6. @uses and @used-by
  • Similar to PHPDoc:@see but different
    • @uses ClassA::MethodName() in Class B implies ClassA::MethodName() is used by Class B in ClassA. Bidirecitonal
    • @see is a one-way link
1.31.4.7. @example

It can be used to demo by presenting the contents of files that use them. 

@example [location] [<start-line> [<number-of-lines>] ] [<description>]
@example example1.php Some Description
1.31.4.8. @property
  • Used when a class or trait implements the 1.34.10.1 and/or 1.34.10.1 magic methods to resolve non-literal properties at run-time
    • ONLY be used above class or trait
  • @property static float $PurchasePrice some more description
  • @property-read and @property-write variants may be used to indicate "magic" properties that can only be read or written
class A implements MyInterface {
    use MyTrait;

    /**
     * @var XYZ
     */
    public $xyz;
}

interface MyInterface {
    const C_1 = 1;

    public function mustBeAbleToDoThis();
}

/**
 * @property XYZ $xyz
 */
trait MyTrait {
    public function mustBeAbleToDoThis() {
        echo $this->xyz->doXYZSubMethod();
    }
}
1.31.4.9. @mixin
  • The mixin DocBlock allows you to signal that all properties and methods of another class are available on the class where the mixin is applied upon.
    • For example, ClassB has __call() function that calls another ClassA and ClassB is not a child/parent of ClassA

  • Example 

    namespace App\Models;

use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Eloquent\Builder;

/**
 * @mixin Builder
 */
class Post extends Model {}
  • Laravel Model Li

1.31.5. datatype 

int
string|bool
class-string<MyClassFQDN> => a class name in string
\Drupal\Core\Database\StatementInterface
int[]
\Drupal\node\NodeInterface[]
$this
static
null
object
resource
true
false
float

Examples 

@return $this
@return static
1.31.5.1. Array syntax, array shape
int[]
[1,2,3]
string[]
[ 'a', 'b' ]
Order[]
[Order1, Order2]
Order[][]
array(array(Order1, Order2), array(Order3,Order4))
array<int,int>
[1=>1, 2=>3]
array<int, string>
[1=>'a', 2=>'b']
array<int|string, Order>
[1=>Order1, 'a'=>Order2]
array<int|string, mixed>
[1=>Order1, 'a'=>123]
array<int, string[]>
[1=>['a']]
array<int, array<int, string>>
[1=> [1=>'a']]
array<string, array<int, string>>
['abc' => [ 1 => 'a' ] ]
array{key1: string, key2: string}
['key1' => 'a', 'key2' => 'b']
array{key1: string, ...}
eq. array{key1: string, ...<array-key, mixed>}
unsealed array
aka open array
(no term)
array<int, array{key1: string, key2: string}>
list<string>
array with key starts from 0 [0 => 'a', 1 => 'b']
(no term)
list<array{key1: string, key2: strin}>

1.31.6. Generic Types / Generics

1.31.6.1. Example
interface EntityAutomobileInterface
{
}

class Car implements EntityAutomobileInterface
{
    public $id;
    public $engineCapacity;

    public function __construct(int $id, float $engineCapacity)
    {
        $this->id = $id;
        $this->engineCapacity = $engineCapacity;
    }
}

// IMPORTANT We can use PhpDoc for Generic Types before PHP natively supports it

/**
 * @template T
 */
interface CollectionInterface
{
    /**
     * @param T $entity
     */
    public function add($entity);

    /**
     * @param int $id
     * @return T
     */
    public function find(int $id);

    /**
     * @return T[]
     */
    public function listAll();
}

// IMPORTANT T of anInterface
/**
 * @template T of EntityAutomobileInterface
 */
interface CollectionAutomobileInterface
{
    /**
     * @param T $entity
     */
    public function add($entity);

    /**
     * @param int $id
     * @return T
     */
    public function find(int $id);

    /**
     * @return T[]
     */
    public function listAll();
}

// IMPORTANT T of aSubClass
/**
 * @template T of Car
 */
interface CollectionCarInterface
{
    /**
     * @param T $entity
     */
    public function add($entity);

    /**
     * @param int $id
     * @return T
     */
    public function find(int $id);

    /**
     * @return T[]
     */
    public function listAll();
}

$car = new Car(1, 1.23);

interface EntityAnimalInterface
{
}

class Monkey implements EntityAnimalInterface
{
    public $name;
    public $gender;

    public function __construct(string $name, float $gender)
    {
        $this->name = $name;
        $this->gender = $gender;
    }
}

$monkey = new Monkey('moneky name', 'f');

// IMPORTANT All good
/** @var CollectionInterface<Car> $carCollection */
$carCollection->find(1)->engineCapacity;

// IMPORTANT All good
/** @var CollectionAutomobileInterface<Car> $carCollection */
$carCollection->find(1)->engineCapacity;

// IMPORTANT No suggestion for `engineCapacity` because `EntityAutomobileInterface` does not have property `engineCapacity`
/** @var CollectionAutomobileInterface<EntityAutomobileInterface> $carCollection */
$carCollection->find(1)->engineCapacity;

// IMPORTANT All good
/** @var CollectionCarInterface<Car> $carCollection */
$carCollection->find(1)->engineCapacity;

// IMPORTANT All good
/** @var CollectionInterface<Monkey> $monkeyCollection */
$monkeyCollection->find(1)->name;

// IMPORTANT All good (PhpStorm static analysis should have thrown some warnings. Tested on PhpStorm 2023.1)
/** @var CollectionAutomobileInterface<Monkey> $monkeyCollection */
$monkeyCollection->find(1)->name;

// IMPORTANT All good (PhpStorm static analysis should have thrown some warnings. Tested on PhpStorm 2023.1)
/** @var CollectionCarInterface<Monkey> $monkeyCollection */
$monkeyCollection->find(1)->name;

// IMPORTANT All good (PhpStorm static analysis should have thrown some warnings. Tested on PhpStorm 2023.1)
/** @var CollectionCarInterface<Monkey> $monkeyCollection */
$monkeyCollection->add($monkey)->name;

1.32. Database

1.32.1. mysqli

  • Driver only for MySQL while php:pdo has 12 different drivers
  • It can be OOP or procedural. php:pdo only has OOP
$mysqli = new mysqli("localhost", "my_user", "my_password", "world");

/* check connection */
if (mysqli_connect_errno()) {
    printf("Connect failed: %s\n", mysqli_connect_error());
    exit();
}

$mysqli->query("CREATE TEMPORARY TABLE myCity LIKE City");

$city = "'s Hertogenbosch";

$city = $mysqli->real_escape_string($city);

if ($mysqli->query("INSERT into myCity (Name) VALUES ('$city')")) {
    printf("%d Row inserted.\n", $mysqli->affected_rows);
}


// You don't have to close it
// $mysqli->close();

$query = <<<SQL
INSERT INTO   abc_inv_tbl (
    ...
  ) VALUES (
    NOW(),
    '{$db->real_escape_string($vehicle_name)}',
    '{$db->real_escape_string($vehicle_cond)}',
    '',
    {$vehicle_year},
    {$vehicle_type},
    {$vehicle_model},
    {$run_hours},
    '{$db->real_escape_string($trans)}',
    '{$db->real_escape_string($horsepower)}',
    '{$db->real_escape_string($desc)}'
  )
SQL;

$con = mysqli_connect("localhost","my_user","my_password","my_db");
// Check connection
if (mysqli_connect_errno()) {
  echo "Failed to connect to MySQL: " . mysqli_connect_error();
}

$sql = "SELECT Lastname,Age FROM Persons ORDER BY Lastname";

if ($result = mysqli_query($con, $sql)) {
  // Seek to row number 15
  mysqli_data_seek($result,14);

  // mysqli_data_seek(mysqli_result $result, int $offset): bool

  // Fetch row
  $row = mysqli_fetch_row($result);

  printf ("Lastname: %s Age: %s\n", $row[0], $row[1]);

  // Loop can also be used while ($row = mysqli_fetch_assoc($result)) {...}

  // Free result set
  mysqli_free_result($result);
}

mysqli_close($con);

1.32.2. PDO

  • It supports named parameters and prepared statements on client side while php:mysqli doesn't

1.33. Function

1.33.1. Anonymous function

  • A function that has no name
  • It may or may not create a closure
    • However, in PHP, any anonymous function is implemented using the Closure class

1.33.2. Closure

  • A function that captures the state of the surrounding (outer) environment and create its own scope
    • However, in PHP, a closure does not inherit the parent scope, it only inherits the global scope and also the $this and self scope
    • JavaScript has 2 types of scopes
      Global
      declared outside of a block
      Local
      declared inside a block
    • JavaScript closure has access to
      • its scope
      • outer function's variables
      • global variables
  • A closure can have a function name in JavaScript. PHP a closure cannot have a name
  • A closure may not be defined as a result of immediately run. See js:closure

  • In PHP, any anonymous function is implemented using the Closure class 

    $message = 'hello';

$example = function ($arg) use ($message) {
var_dump($arg . ' ' . $message);
};

// Inherit by-reference
$example = function () use (&$message) {
var_dump($message);
};
$example();

// Very different from JavaScript's Closure
$create_printer = function () {
$my_number = 42;

// First, `use` has to be used
return function () use ($my_number) {
    $my_number++; // can only modify $my_number inside the closure
    echo "My fav number is $my_number\n";
};
};

$my_printer = $create_printer();
$my_printer(); // My fav number is 43
$my_printer(); // My fav number is 43
  • Callable type
  • Cannot use
    • superglobals,
    • $this
    • variables with the same name as a parameter
    • an array element $array['a'], have to use the whole array
1.33.2.1. Closure Recursive

  • An anonymous function or a closure can call itself 

    $f = function() use($bar, $foo, &$f) {
   $f();
};

$arr = ($_f = function ($argument) use (&$_f) {
    if (count($argument) < 3) {
        $argument[] = 'foo';
        $argument = $_f($argument);
    }
    return $argument;
})(
    array()
);

var_dump($arr === ['foo', 'foo', 'foo']);

1.33.3. Callable

  • https://www.php.net/manual/en/language.types.callable.php
  • Static class method
    array('MyClass', 'myCallbackMethod')
    call_user_func(array('MyClass', 'myCallbackMethod'));
    • FQDN
    (no term)
    [MyClass::class, 'myCallbackMethod']
    • FQDN or alias
    (no term)
    MyClass::myCallbackMethod
    • call_user_func('MyClass::myCallbackMethod');
    • $fn = 'MyClass::myCallbackMethod'; $fn('param1');

      • Use FQDN. Alias cannot be used! 

        namespace MyNamespace;

use ThirdPartyUtils as Utils;

class Example {
    public static example() {
        $fn = 'Utils::output';
        // error
        echo $fn('param1');
    }
}
  • array($obj, 'myCallbackMethod')

  • Named function 

    function named($a) {
    var_dump('This is ' . $a);
}

$unnamed = function ($a) {
    var_dump('This is ' . $a);
};

function caller($callee, string $name) {
    is_callable($callee) === true;
    $callee($name);
}

caller('named', 'named');
caller($unnamed, 'unnamed');

1.33.4. Return by reference

// Chaining
function abc() {
    return ['a' => 123];
}

abc()['a'];

// Return a reference to the value
class Config {
    private $values = [];

    // return a REFERENCE to the actual $this->values array
    public function &getValues() {
        return $this->values;
    }

    // don't need to set function to pass by reference as objects are passed by reference by default, other data types are passed by value
    public function getObjectValue() {
        return $this->values;
    }

}

$config = new Config();
$config->getValues()['test'] = 'test'; // mutate a private property from outside
var_dump($test = $config->getValues()['test'], $test === 'test');

1.33.5. Lambda function

  • A function can be passed as data to e.g. functions, array values 

    // Lambda function is available since PHP 5.3.0
$func = function($value) {
    return $value * 2;
};
print_r(array_map($func, range(1, 5)));

  • Use array to pass object/class static/non-static methods as lambda 

    $a = [
    'item1' => ['A', 'abc2'], // abc2 can be either static or non-static.
    // But abc2 method is run on static context which means if abc2 is non-static and `$this` is used inside
    // it will fail
    // Also calling non-static (instance) method in static context throws a warning
    // with namespace ['\LlPassObjectMethods\A', 'abc2'] or ['LlPassObjectMethods\A', 'abc2']
    'item2' => [$this, 'abc3'], // An instance's method is run
];
$a['item1']('Pass self instance method in static context');
$a['item2']('Pass self instance method in instance context');

  • Lambda function defined from a static function However, PhpStorm can't track the usage of that static method 

    $fn = "Foo::staticMethod"; // can be single quote
$fn(123);

  • Not necessarily anonymous e.g. in JavaScript, the function passed in may have a function name 

    $('#el').on('click', function clickHandler () {
  console.log(`
  This is an example of a lambda expression that is not anonymous. As you can
  see, it clearly has a name, clickHandler, which can be used inside the
  function for the purpose of recursion (also an important concept in functional
  programming).

  It is a lambda expression because of the semantic use -- it's being passed
  to another function as data. The .on() function is using the function as
  an argument -- in other words, communicated as a message (i.e. functions as
  data).
  `);
});

1.33.6. Arrow function

  • https://www.php.net/manual/en/functions.arrow.php
  • A shorthand to define a function. As far as I know in all languages
    • it's anonymous function
    • called short closures in PHP
  • Since PHP 7.4
  • Only one expression in the function body with no curly brackets
  • Don't need to use use
  • See array_map

  • Example 

    $y = 1;

// This is valid for PHP 7.4+. Not valid for 7.3
var_dump(
    (fn($x) => $x + $y)(2)
    === 3
);

var_dump(
    (function ($x) use ($y) {
        return $x + $y;
    })(
        2
    )
    === 3
);

1.33.7. IIFE - Immediately Invoked Function Expressions

1.33.7.1. v7.0+
echo (function() {
return 42;
})();

$foo = (function() {
return function($a) {
    return $a + 42;
};
})();

echo $foo(10); // 52

$foo = (function($a) {
return function($b) use ($a) {
    return $a + $b;
};
})(42);
echo $foo(10); // 52


// Pass by reference
$sum = 0;
(function ($a, $b, &$sum) {
    $sum = $a + $b;
    echo $sum . ' inside IIFE' . PHP_EOL;
})(
    25, 9, $sum
);
var_dump($sum); // 34
1.33.7.2. Below PHP 7.0
call_user_func(function() {
echo "Hello World!";
});

call_user_func(function($a, $b) {
$sum = $a + $b;
echo $sum;
}, 25, 9);

// Or
$a = 25; $b = 9;
call_user_func(function() use ($a, $b) {
$sum = $a + $b;
echo $sum;
});

// Pass by reference :: use call_user_func_array
call_user_func_array(
    function ($a, $b, &$sum) {
        $sum = $a + $b;
        echo $sum . ' inside IIFE' . PHP_EOL;
    },
    array(25, 9, &$sum)
);

1.33.8. func_get_args, func_get_arg 

function foo() {
    var_dump(func_num_args() === 3);
    var_dump(func_get_arg(1) === 2);
    var_dump(func_get_arg(10) === false);
    var_dump(($eleventhArgValue = func_num_args() >= 10 ? func_get_arg(10) : null) === null);
    var_dump(func_get_args() === array(1, 2, 3));
}

foo(1, 2, 3);

1.33.9. ReflectionFunction 

$reflFunc = new ReflectionFunction('my_function');

// multiple files declare the same my_function function. To find out which file the loaded function is loaded from:
// file absolute path is shown
var_dump($reflFunc->getFileName() . ':' . $reflFunc->getStartLine());

1.34. OOP

1.34.1. Sample

RestJson.php 

$rest = new RestJson('12345', 'trucks', 'new');
// Class name should use UpperCamel, can have underscores and numbers
// SampleXmlClass, not SampleXMLClass
// Names should not include Drupal, Class
// Interface should be always suffixed "Interface"

// class names and function names are case-insensitive. However, class variable names are case sensitive

class RestJson {

    // Constant
    // - defined at compile time not at runtime
    // - strings, booleans, integers, or another defined constant. Can be array for PHP 5.6.0+
    // - Can be redefined in child class
    // - interface can also have constants
    // - default visibility of constants is public, can be set (even public) for PHP 7.1.0+
    // - For PHP 5.6+, arithmetic calculation is possible

    const MY_CONSTANT_VAR = 1;
    const MY_CONSTANT_VAR_2 = MY_CONSTANT_VAR;
    const B_CONSTANT = B::FROM_ANOTHER_CLASS;
    // usage self::ABC (self class or this class), parent::ABC, static::ABC (called class)
    // self:: eq. __CLASS__, get_class(), self::class
    // static:: eq. get_called_class()
    // RestJson::ABC
    // $className = 'RestJson'; $className::ABC

    // Class properties
    // - Only primitive types e.g. int, string, number, array
    // - PHP version > 5.4, arithmetic calculation is allowed
    // array cannot contain expressions e.g. lambda functions
    // get_class_vars(__CLASS__) == array('property1Name' => 'property1Value', 'privateStaticPropertyName' => 123)

    public static $valid_address_types = [
        RestJson::MY_CONSTANT_VAR => 'Residence',
    ];

    // static property cannot be called $this->staticProp!
    // Has to be RestJson::$staticProp or
    // $class = get_class($instance); $class::$staticProp

    public static $filterables;
    // singleton. same for every instance.
    //
    // Can only be updated by static function/method
    // Static properties can be accessed in these ways (have to use :: but not ->)
    // - Inside class: self::$a, parent::$a
    // - Outside class:
    //   - RestJson::$staticProperty
    //   - $aClassInstance::$staticProperty
    //   - $className = 'RestJson'; $className::$staticProperty
    //   - $this->caller::$staticProperty :: valid but phpStorm reports an error. In order for phpStorm to show the usage, need to use RealClassName::$staticProperty

    public $lwp_options, $listings;
    // Object properties
    // - variable and function names should be lowerCamel and avoid underscores
  // - variable can have default value that is static. e.g. value of time() is not allowed or reference to another varialbe $a = $b
    // - Dynamic refer to an object property
    //   - `$obj->$a` where `$a === 'lwp_options';`
    //   - `$obj->$array[0]` where `$array[0] === 'lwp_options'`

    public function __construct($did, array $industry, $condition) {
        // Call a static function
        $this->lwp_options = self::getLwpOptions();
        // Call a instance function
        $this->listings = $this->getListings();
    }

    public static function getLwpOptions() {
        // A static function can return a value
        // can change a static property
        // self::$filterables[] = '123';
        // But can't access the instance $this
        // Can't modify instance value $this->aPropertyName
        // Can't call instance method $this->aMethod();
    }

// static methods can be accessed in these ways (both :: and -> can be used)
// - Inside class
//   - RestJson::staticMethod();
//   - $this->staticMethod();
// - Outside class
//   - RestJson::staticMethod();
//   - $aClassInstance->staticMethod();
//   - $className = 'RestJson'; $className::$staticMethod

// Private and protected properties/methods prefixed with a single underscore
// Private and protected cannot be used inside `include` `include_once`
}

// Class::aStaticMethod()
// Class::$aStaticProperty

1.34.2. Instantiate Dynamically

<?php

class A {
    public $name = '';

    public function __construct(string $property = '') {
        $this->name = $property;
    }

    public function getName() {
        return $this->name;
    }

    public static function getInstance($property): self {
        return new A($property);
    }

    public static function getInstanceNames($property): array {
        return array(
            (new A($property))->getName() === $property,
            (new static($property))->getName() === $property,
            (new static())->getName() === '',
            (new static)->getName() === '',
            (new self($property))->getName() === $property,
            (new self())->getName() === '',
            (new self)->getName() === '',
        );
    }

}

$className = 'A'; // namespace ::  `$className = 'MyNamespace\A';`
// Since PHP 7
$instance = new $className('abc'); // `$instance = new (A::class);`

// `A::class` returns string of the fully qualified class name

// Below PHP 7
$instance = call_user_func([$className, 'getInstance'], 'abc');

$instance->getName();

// In one line
$property = (new $className('abc'))->getName();

var_export(A::getInstanceNames('xyz'));

1.34.3. OOP Features

  • Abstraction
  • Encapsulation (expose functionality and access control)
    protected
    accessed within class, inherited (parent) / inheriting (child) classes
    private
    only the current class can access
    (no term)
    prefix underscore for protected and private properties' names
  • inheritance
  • accomplish one task
  • interface interact with classes without knowing what class it is

1.34.4. Inheritance

  • Child can override parent's constants but not constants declared in interface
  • Child can override parent's methods but with the same name and arguments (except constructors)
  • Child can override parent's methods and make them have the same or less restricted visibility
// class ParentChild extends Parent {}
class AddressResidence extends Address {}
1.34.4.1. Flaws of inheritance
  • Inheritance achieves 2 things
    • Re-use code and further extend
    • child classes must have all of what their parent class has
    • Pros
      • Good for if you have 100 child classes
    • Cons
      • Expensive to refactor if it's desired to remove methods from parent class and create a new parent class
  • The other alternative is using composition with dependency injection (with Interface)
    • Pros
      • Reduce coupling to re-used code
      • Adaptable as new requirements come in
    • Cons
      • A lot of instantiations for interface implementations in Dependency Injection
      • A lot of wrappers to expose methods/properties injected by Dependency Injections in order for the child class to do what ever the injected object can do from public
      • Don't use Composition + Dependency Injection in these scenario
        • Middlewares which can be chained

1.34.5. Final keyword

  • It prevents child classes from overriding a method or constant (as of PHP 8.1.0) by prefixing the definition with final.
  • If the class itself is being defined final then it cannot be extended.

1.34.6. Abstract Class

  • Abstract class cannot be instantiated
    • It's used as a base to make child classes and the base can have abstract methods that child must declare
  • If a class has an abstract method, then the class itself must be abstract
    • If a method is abstract, any class (not abstract class) that extends that abstract class containing the method must also declare a method with the same name and arguments
  • No abstract property (static nor non-static)
abstract class Address {
  public function __construct() {
    $this->_init();
  }

  /**
   * Force extending classes to implement init method.
   */
  abstract protected function _init();
}

class AddressBusiness extends Address {
  // Methods that implement abastract method
  // must have the same scope OR less restrictive
  protected function _init() {
    $this->_setAddressTypeId(Address::ADDRESS_TYPE_BUSINESS);
  }
}
1.34.6.1. Force inherited classes to define a constant
abstract class Foo {
  // Self-referential 'abstract' declaration
    // removing abstract from abstract clss Foo and changing self::NAME to static::NAME will not work
    // although on PHPStorm syntax highlighting shows there's an error, but it's totally fine
    const NAME = self::NAME;
}

class Fooling extends Foo {
    // Overrides definition from parent class
  // Without this declaration, an error will be triggered
  const NAME = 'Donald';
}

$fooling = new Fooling();

echo $fooling::NAME;
1.34.6.2. Enummeration 
abstract class BasicEnum {

    private static $constCacheArray = null;

    private static function getConstants() {
        if (self::$constCacheArray == null) {
            self::$constCacheArray = array();
        }
        $calledClass = get_called_class();
        if ( ! array_key_exists($calledClass, self::$constCacheArray)) {
            $reflect = new ReflectionClass($calledClass);
            self::$constCacheArray[$calledClass] = $reflect->getConstants();
        }

        return self::$constCacheArray[$calledClass];
    }

    public static function isValidName($name, $strict = false) {
        $constants = self::getConstants();

        return ($strict) ? array_key_exists($name, $constants) :
            in_array(strtolower($name), array_map('strtolower', array_keys($constants)));
    }

    public static function isValidValue($value, $strict = true) {
        return in_array($value, array_values(self::getConstants()), $strict);
    }

    /**
     * @param array $values
     * @param bool $strict default true
     * @param bool $returnInvalidValues default false, return bool
     *
     * @return array|bool default bool, or array of invalid values with original key
     */
    public static function areValidValues($values, $strict = true, $returnInvalidValues = false) {
        if ( ! is_array($values)) {
            return false;
        }
        $allValues = array_values(self::getConstants());
        $invalidValues = array();
        foreach ($values as $value) {
            if ( ! in_array($value, $allValues, $strict)) {
                if ( ! $returnInvalidValues) {
                    return false;
                }
                else {
                    array_push($invalidValues, $value);
                }
            }
        }

        return ($returnInvalidValues) ? $invalidValues : true;
    }

}

abstract class DaysOfWeek extends BasicEnum {
    const Sunday = 0;
    const Monday = 1;
    const Tuesday = 2;
    const Wednesday = 3;
    const Thursday = 4;
    const Friday = 5;
    const Saturday = 6;
}

var_dump(DaysOfWeek::Monday);
var_dump(DaysOfWeek::isValidName('Monday'));
var_dump(DaysOfWeek::isValidValue(1));
var_dump(DaysOfWeek::areValidValues(array(0, 1, 100)));

1.34.7. Interface

  • Only 3 things an interface can do

    1. Define public method stubs or public methods signatures without bodies

      • Specify what methods must be implemented but not how
      class CustomerQuery implements CustomerQueryInterface {
  public function whereId($id) {
    // do something to get Customer by $id
  }

  public function whereType($type) {
    // do something to get Customer by $type
  }
}
      • A class can implement multiple interfaces
      • Use final in parent class to prevent child class from overriding the method/property
      final public function load() {}

    2. Extend one or more interfaces 

      interface QueryInterface {
  public function whereId($id);
}

interface CustomerQueryInterface extends QueryInterface {
  public function whereType($type);
}
    3. Constants can be defined in Interface. However, prior to PHP 8.1.0, they cannot be overriden by any class/interface that inherits the Interface
    4. Interface may include static methods. However, these static methods are not required to be implemented
  • No State!
    • Interface cannot have properties
    • To declare individual static properties for subclasses
      • A subclass without declaring a static property which the parent class declares, the subclass' static property uses and modifies the parent class' static property
      • Don't use 1.34.9 for declaring static properties where the same static properties are declared in parent class
  • Used for dependency injection
    • CustomQueryInterface (CQI) is an interface building on top of QueryInterface, any implementation of CQI can be passed as dependency injection to constructor of another class
  • A class can implement a method with the same name in multiple interfaces. As long as the function signature matches one interface. But only PHP 7.4+ supports it.

1.34.8. Interface vs Abstract Class

  • Interfaces
    • No state (cannot have properties)
    • Cannot implement another interface
    • Interface methods must be public and they have no method body
    • a class that implements an interface must provide an implementation of all the methods of the interface
      • Static methods in interface are not required to be implemented
  • abstract classes
    • may contain state (data members) and/or implementation (methods)
    • can be inherited without implementing the abstract methods
  • Common
    • can extend a parent
  • Biggest difference
    • A class may implement multiple interfaces
    • A class can only extend one abstract class

1.34.9. Trait

  • Always use a trait that fulfils one, all or extra capabilities that are
    • required by an interface that the current class implements
    • or required by an abstract class that the current class extends to
  • A trait can
    • Use other traits
    • Define properties (static and non-static)
  • Classes use a trait
    • Can override a trait's method

    • Can extend a trait's method 

      trait FooTrait {
    public function bar() {
        return 'A';
    }
}

class Example {
    use FooTrait {
        bar as traitBar;
    }

    public function bar() {
        return $this->traitBar() . 'B';
    }
}

echo (new Example())->bar(); // Outputs 'AB'
  • Limitation
    • Cannot extend a class
    • Cannot implement an interface
    • Constants are only allowed since PHP 8.2

    • Calling a static method or accessing a static property directly on a trait is deprecated since PHP 8.1 

      trait TestTrait {
    public static function foo() {
        echo "Foo\n";
    }

    public static function bar() {
        self::foo();
    }

}

// Works until PHP 8.1 (throws warning Deprecated)
TestTrait::foo();

class A {
    use TestTrait;
}

// Works in all versions
A::bar();
1.34.9.1. Example
interface PersonInterface {
  const C_1 = 123;

    public function greet();
    public function eat($food);
    public function drink($drink);
}

trait EatingTrait {

    // Traits can define static variables, static methods and static properties.

    public $traitProperty = true; // the class uses the trait cannot define a property with the same name unless it has the same visibility and initial value
    public function eat($food) {
        $this->putInMouth($food);
    }

    public function drink($drink) {
        echo "Trait drinks {$drink}" . PersonInterface::C_1;
    }

    private function putInMouth($food) {
        echo "Trait eats {$food}";
    }
}

class NicePerson implements PersonInterface {
    // use EatingTrait;
    // multiple traits
    /*
    use A, B {
        B::smallTalk insteadof A; // use B's smallTalk instead of A's smallTalk to avoid same method conflict
        A::bigTalk insteadof B;
        B::bigTalk as talk; // change B's bigTalk method name to talk
    }
    */

    use EatingTrait {
        drink as protected traitDrink; // also change method visibility
    }

    public function greet() {
        echo 'Good day, good sir!';
    }

    public function drink($drink) {
        $this->traitDrink("super {$drink}");
    }
}

class MeanPerson implements PersonInterface {
    use EatingTrait;

    public function greet() {
        echo 'Your mother was a hamster!';
    }
}

$nicePerson = new NicePerson();
$nicePerson->drink('Kool-Aid'); // Trait drinks super Kool-Aid

1.34.10. Magic Methods

  • http://php.net/manual/en/language.oop5.magic.php
  • Special methods in PHP classes
  • Starts with 2 underscores. They are for:
    • Trigger custom behavior
    • Customize object creation
    • access or change hidden properties
  • With magic method, the performance is about 3 to 10 times slower..
1.34.10.1. Overloading
  • These are to dynamically "create" a property or method of an instance that has not been declared, or is not visible in the current scope.
  • Can't be used in static properties.
  • triggered when writing data to inaccessible properties
  • public function __get(string $name): mixed is triggered when reading data from inaccessible properties
  • They are all public functions
protected $_postal_code;

function __get($name) {

  if (!$this->_postal_code) {
    $this->_postal_code = $this->_postal_code_guess();
  }
  $protected_property_name = '_' . $name;
  if (property_exists($this, $proteced_property_name)) {
   return $this->$protected_property_name;
  }

  trigger_error('Undefined property via __get:' . $name);
  return NULL;
}
1.34.10.2. __construct, __destruct
class BaseClass {
  function __construct() {
    print "In BaseClass constructor\n";
  }
}

class SubClass extends BaseClass {
  function __construct() {
    parent::__construct();
    print "In SubClass constructor\n";
  }
}
1.34.10.3. __toString()

Turn an object to a string when echo $obj; Exception cannot be thrown within __toString() method.

1.34.10.4. __clone()
class SubObject
{
  static $instances = 0;
  public $instance;

  public function __construct() {
    $this->instance = ++self::$instances;
  }

  public function __clone() {
    $this->instance = ++self::$instances;
  }
}

class MyCloneable
{
  public $object1;
  public $object2;

  function __clone()
  {
    // Force a copy of this->object, otherwise
    // it will point to same object. (shallow copy)
    $this->object1 = clone $this->object1;
  }
}

$obj = new MyCloneable();

$obj->object1 = new SubObject();
$obj->object2 = new SubObject();

$obj2 = clone $obj;


print("Original Object:\n");
print_r($obj);

print("Cloned Object:\n");
print_r($obj2);

/*
Original Object:
MyCloneable Object
(
  [object1] => SubObject Object
(
  [instance] => 1
    )

    [object2] => SubObject Object
(
  [instance] => 2
    )

)

Cloned Object:
MyCloneable Object
(
  [object1] => SubObject Object
(
  [instance] => 3
    )

    [object2] => SubObject Object
(
  [instance] => 2
    )

)
*/
1.34.10.5. __set()
  • It's run when writing data to inaccessible (protected or private) or non-existing properties
public __set(string $name, mixed $value): void

1.34.11. Magic Constants 2

1.34.12. is_object, is_a, is_subclass_of, get_class(), get_parent_class(), get_called_class()

get_class(object $object = ?): string
get_parent_class(object|string $object_or_class = ?): string

<?php

class A {
}

class B extends A {
}

class C extends B {
}

$a = new A();
$b = new B();
$c = new C();

var_dump(is_a($b, 'A') === true);
var_dump(is_a($b, 'B') === true);
var_dump(is_a($a, 'A') === true);

var_dump(get_class($b) === 'B');

var_dump(is_subclass_of($b, 'A') === true);
var_dump(is_subclass_of($a, 'A') === false);
var_dump(is_subclass_of($c, 'A') === true);
var_dump(is_subclass_of($c, 'B') === true);

var_dump(get_parent_class($b) === 'A');
var_dump(get_parent_class($a) === false); // $a has no parent class! Returns false

// inside a class, get_class() and get_parent_class() don't need to provide a parameter
1.34.12.1. get_called_class()
namespace Lili;

class foo {

    static public function testGetCalledClass(): string {
        // FQDN. eq. to:
        // return static::class;
        // return (new \ReflectionClass(get_called_class()))->getName();
        return get_called_class();
    }

    static public function testGetClass(): string {
        // FQDN. eq. to:
        // return __CLASS__;
        // return self::class;
        // return (new \ReflectionClass(get_class()))->getName();
        return get_class();
    }

    static public function testReflectionGetShortName(): string {
        // eq.
        // return (new \ReflectionClass(get_called_class()))->getShortName();
        return substr(strrchr(get_called_class(), '\\'), 1);
    }

}

class bar extends foo {
}

var_dump(
    foo::testGetCalledClass() === 'Lili\foo',
    bar::testGetCalledClass() === 'Lili\bar',
    foo::testGetClass() === 'Lili\foo',
    bar::testGetClass() === 'Lili\foo',
    foo::testReflectionGetShortName() === 'foo',
    bar::testReflectionGetShortName() === 'bar'
);

1.34.13. Clone, copy and reference

$obj1 == $obj2
do 2 objects have the same property names and values
$obj1 === $obj2
do 2 objects have the same properties and also are those 2 instances of the same class
get_class($obj1)
$obj1 instanceof AddressBusiness
$obj2 = $obj1;
assign $obj1 to $obj2 and both objects refer to the same object in memory. Changing properties of $obj2 will also change $obj1.
  • Reference is faster than clone.
  • $obj2 === $obj1 is true
$obj2 = clone $obj1;
create a shallow copy and $obj2 is a new object in memory. The class of $obj1 may run __clone() magic method if it exists.
$a = new stdClass();

$a->name = 'Li';
$bindOnce = $a;
$bindAlways = &$a;
$clone = clone $a;

var_dump(($a === $bindOnce) ===  true);
var_dump(($a === $bindAlways) ===  true);
var_dump(($a === $clone) ===  false);

// Now $a points to a new memory in place (a different object)
$a = new stdClass();
$a->name = 'name';

var_dump(($a === $bindOnce) === false);
var_dump(($a === $bindAlways) ===  true);
var_dump(($a === $clone) ===  false);

1.34.14. Autoload Classes

  • Autoloading is not available if using PHP in CLI interactive mode
  • spl_autoload_register Parameters
    $autoload_function

    register this to __autoload() functions

    • If no parameter is provided spl_autoload_register(), then this is the default behavior
    // Your custom class dir
define('CLASS_DIR', 'class/')

// Add your class dir to include path
set_include_path(get_include_path().PATH_SEPARATOR.CLASS_DIR);

// You can use this trick to make autoloader look for commonly used "My.class.php" type filenames
spl_autoload_extensions('.class.php'); // or ".php,.inc"

// Use default autoload implementation which is
//spl_autoload ( string $class_name [, string $file_extensions = spl_autoload_extensions() ] ) : void

spl_autoload_register();

// Later on..
use My\Name\Object
// it will map to "class/My/Name/Object.class.php" file path!
    • Not necessary to use include_once or require_once as the class is for sure missing
    $throw
    whether to throw exceptions when $autoload_function cannot be registered
    $prepend
    prepend $autoload_function to the autoload queue instead of appending it
  • Use spl_autoload_functions(); to get all registered __autoload() functions
  • class_exists('aNonExistingClass') triggers all spl_autoload_register(...)
spl_autoload_register ([ callable $autoload_function [, bool $throw = TRUE [, bool $prepend = FALSE ]]] ) : bool

function my_autoloader($class) {
    include 'classes/' . $class . '.class.php';
}

spl_autoload_register('my_autoloader');

// Or, using an anonymous function as of PHP 5.3.0
spl_autoload_register(function ($class) {
    include 'classes/' . $class . '.class.php';
});

// Use the function to just include files (variable, function, class definition).
// Pass other variables to the function
$someNonGlobalVariable = '';

spl_autoload_register(
    function ($class) use ($someNonGlobalVariable){
        $map = array(
            'Dependency_function_a' => __DIR__ . '/../../app/inc/liba.inc.php',
            'Dependency_function_b' => __DIR__ . '/../../app/inc/functionb.inc.php',
            'NormalClass' => __DIR__ . '/../../app/classa/ClassA.php',
            'MyNamespace\ClassC' => __DIR__ . '/../../app/classc/ClassC.php',
        );
        if (isset($map[$class])) {
            $functionName = explode('Dependency_', $class);
            if (isset($functionName[1])) {
                if ( ! function_exists($functionName[1])) {
                    include_once $map[$class];
                }
                eval("class $class {}");
            }
            else {
                include_once $map[$class];
            }
        }
    }
);

new Dependency_function_a();
function_a();

// liba.inc.php
echo $someNonGlobalVariable;

namespace Foobar;

class Foo {
    static public function test($name) {
        print '[['. $name .']]'; // [[Foobar\InexistentClass]] 
    }
}

spl_autoload_register(__NAMESPACE__ .'\Foo::test'); // As of PHP 5.3.0

1.34.15. Design Pattern

1.34.15.1. Singleton (Creational DP)

e.g. database connection

1.34.15.2. Prototype (Creational DP)
  • Object clone
1.34.15.3. Builder (Creational DP)
  • Query builder is a good example
  • Hierarchy
    • Director
      • Builders
        • products
1.34.15.4. Lazy Initialization
  • at the beginning, properties are null and populated on demand
  • If null, set and return
  • If value, just return
1.34.15.5. Factory Method (Creational DP)
  • Create objects without specifying class
    • Dynamically create instantiation
    • Good when instantiation process is complicated and needs to be abstracted into the factory
class Address {
  const ADDRESS_TYPE_RESIDENCE = 2;

  static public $valid_address_types = array(
    Address::ADDRESS_TYPE_RESIDENCE => 'Residence',
    Address::ADDRESS_TYPE_BUSINESS => 'Business',
    Address::ADDRESS_TYPE_PARK => 'Park',
  );

  final public static function getInstance($address_type_id, $data = []) {
    $class_name = 'Address'. self::$valid_address_types[$address_type_id];
    if (!class_exists($class_name)) {
        throw new ExceptionAddress(
            'Address subclass not found. cannot create: ',
            self::ADDRESS_ERROR_UNKNOWN_SUBCLASS
        );
    }
    return new $class_name($data);
  }
}

$address_residence = AddressFactory::getInstance(Address::ADDRESS_TYPE_RESIDENCE);
1.34.15.6. Facade (Structural DP)
  • Sample
    • Download a YouTube video has a lot of steps which can be done by using 2 objects YouTube and FFMpeg
      • Use YouTube to get the title and then the video file
      • Use FFMpeg to resize the video
      • Use FFMpeg to extract the thumbnail
      • Use FFMpeg to convert to different formats and save the video files
    • Create a new class YouTubeDownloader, instantiate it with objects YouTube and FFMpeg, and a method called downloadVideo to call the object methods.
  • It creates an abastract level where details of YouTube and FFMpeg do not need to be exposed
    • It creates a new interface for existing objects
1.34.15.7. Proxy (Structural DP)
  • Sample 1
    • The original object (a service) is massive in memory, you don't want to keep it for long. It has all the methods but they are not all needed all the time.
      • Other ways
        • Lazy initialization
          • But the object might be created by 3rd party which you have no control of.
    • Create a new class which only instantiates the original object when needed (collect requirements before request), calls partial methods of the original object (performs the service), modifies or cache the results (after the service/request), and destroys the original object
1.34.15.8. Dependency Injection

  • Dependencies are given rather than creating them. So instead of 

    class CustomersController {
  private $customerQuery;

  public function __construct() {
$this->customerQuery = new CustomerQuery();
  }
}

    We use this, where $customerQuery is an instance of CustomerQueryInterface 

    class CustomersController {
  private $customerQuery;

  public function __construct(CustomerQueryInterface $customerQuery) {
$this->customerQuery = $customerQuery;
  }
}

  • Example 

    class StoreService {
private $geolocationService;

public function __construct(GeolocationService $geolocationService) {
    $this->geolocationService = $geolocationService;
}

public function getStoreCoordinates($store) {
    return $this->geolocationService->getCoordinatesFromAddress($store->getAddress());
}
}

    interface GeolocationService {
    public function getCoordinatesFromAddress($address);
}

class GoogleMaps implements GeolocationService { }

class OpenStreetMap implements GeolocationService { }
1.34.15.9. Decorator (Structural DP)
  • Decorators can be stacked and executed all together
    • Concrete Decorators (MarkdownFormat, DangerousHTMLTagsFilter) extend a basic Decorator (TextFormat)
      • The basic Decorator (TextFormat)
        • implements an interface (InputFormatInterface) which contains methods that the other Decorators must implement (formatText)
      • Other concrete Decorators override the parent's method (formatText) by calling the parent's method and its own code
  • https://refactoring.guru/design-patterns/decorator/php/example#example-1
  • Very similar to Chain of Responsibility (CoR). However, there are crucial differences
    • CoR can stop passing the request further

Class Book has 3 methods: getAuthor, getTitle, getAuthorAndTitle

Decorator BookTitleDecorator adds methods: resetTitle and showTitle. When it instaniates, it resets the title of an instance of Book (in this example, the original title is not changed) Decorator BookTitleStarDecorator extends BookTitleDecorator and adds method starTitle which outputs title in a different format Decorator BookTitleExclaimDecorator extends BookTitleDecorator and adds metod exclaimTitle which outputs title in another different format

Decorator adds a wrapper on top of the original class Book with extra functions. This way it doesn't change other instances of the Book class.

Because the base decorator BookTitleDecorator has dependency injection of class Book, it also has Dependency Injection design pattern.

Moreover, a decorator has to know which class the decorator decorates. But this class can be abstract (superclass). 

$patternBook = new Book('Gamma, Helm, Johnson, and Vlissides', 'Design Patterns');
$decorator = new BookTitleDecorator($patternBook);
$starDecorator = new BookTitleStarDecorator($decorator);
$exclaimDecorator = new BookTitleExclaimDecorator($decorator);

writeln('showing title : ');
writeln($decorator->showTitle());
writeln('');

writeln('showing title after two exclaims added : ');
$exclaimDecorator->exclaimTitle();
$exclaimDecorator->exclaimTitle();
writeln($decorator->showTitle());
writeln('');

writeln('showing title after star added : ');
$starDecorator->starTitle();
writeln($decorator->showTitle());
writeln('');

writeln('showing title after reset: ');
writeln($decorator->resetTitle());
writeln($decorator->showTitle());
writeln('');

writeln('END TESTING DECORATOR PATTERN');

function writeln($line_in) {
    echo $line_in."<br/>";
}


class Book {
    private $author;
    private $title;
    function __construct($title_in, $author_in) {
    $this->author = $author_in;
    $this->title  = $title_in;
    }
    function getAuthor() {
    return $this->author;
    }
    function getTitle() {
    return $this->title;
    }
    function getAuthorAndTitle() {
  return $this->getTitle().' by '.$this->getAuthor();
    }
}

class BookTitleDecorator {
    protected $book;
    protected $title;
    public function __construct(Book $book_in) {
    $this->book = $book_in;
    $this->resetTitle();
    }   
    //doing this so original object is not altered
    function resetTitle() {
    $this->title = $this->book->getTitle();
    }
    function showTitle() {
    return $this->title;
    }
}

class BookTitleExclaimDecorator extends BookTitleDecorator {
    private $btd;
    public function __construct(BookTitleDecorator $btd_in) {
    $this->btd = $btd_in;
    }
    function exclaimTitle() {
    $this->btd->title = "!" . $this->btd->title . "!";
    }
}

class BookTitleStarDecorator extends BookTitleDecorator {
    private $btd;
    public function __construct(BookTitleDecorator $btd_in) {
    $this->btd = $btd_in;
    }
    function starTitle() {
    $this->btd->title = Str_replace(" ","*",$this->btd->title);
    }
}
1.34.15.10. Adapter (Structural DP)
  • Allow objects with incompatible interfaces to collaborate
1.34.15.11. Strategy Pattern (Behavioral DP)
  1. Example 1
    • Finite number of strategies, check each one if it can be satisfied in a loop, the last strategy is the best preferred one
    • Each strategy implements a common interface which has a check method: isAvailable and a action-method: display
    • Each strategy is singleton
    • The best strategy is chosen in an Lazy Initialization process
    class Address {
  const ADDRESS_ERROR_NO_DISPLAY_STRATEGY = 1002;

  private static $_display_strategies = [
    'AddressDisplayNoCountry', 'AddressDisplayFull', 'AddressDisplayPark'
  ];

  private $_display_strategy;

  function display() {
    // Lazy initialization
    if (is_null($this->_display_strategy)) {
  foreach (self::$_display_strategies as $strategy_class_name) {
    if ($strategy_class_name::isAvailable($this)) {
      $this->_display_strategy = $strategy_class_name;
    }
  }
    }
    if (!$this->_display_strategy) {
  throw new Exception('No display strategy found', self::ADDRESS_ERROR_NO_DISPLAY_STRATEGY);
    }
    $display_strategy = $this->_display_strategy;
    return $display_strategy::display($this);

  }

}

    interface AddressDisplay {
  /**
   * AddressDisplay an Address.
   * @return string
   */
  public static function display($address);

  /**
   * Is this method of display available?
   * @return boolean
   */
  public static function isAvailable($address);
}

    class AddressDisplayFull implements AddressDisplay {
  /**
   * Display an addess with a country.
   */
  public static function display($address) {
    $output = AddressDisplayNoCountry::display($address);
    $output .= '<br/>';
    $output .= $address->country_name;
    return $output;
  }

  /**
   * Is this method of display available?
   * @return boolean
   */
  public static function isAvailable($address) {
    return $address->country_name ? TRUE : FALSE;
  }
}
  2. Example 2
    // The following is not ideal

class FetchMode 
{
 const FetchAll = 0;
 const FetchOne = 1;
 const FetchRow = 2;
}

// in another file
public function getRecordSet($mode)
{
    switch ($mode) {
        case FetchMode::ALL:
            // code to do a fetchAll
            break;
        case FetchMode::ONE:
            // code to do a fetchOne
            break;
        default:
    }
}

// Let's refactor
public function getRecordSet(FetchMode $fetchModeStrategy)
{
    return $fetchModeStrategy->fetch();
}

interface FetchMode
{
    public function fetch();
}

class FetchOne implements FetchMode
{
    public function fetch()
    {
        // code to fetchOne
    }
}

class FetchAll
{
    // ...
}

class FetchRow
{
    // ...
}
1.34.15.12. Iterator (Behavioral DP)
1.34.15.13. Observer Pattern (Behavioral DP)
  • Provide a way to subscribe and unsubscribe to and from events for any object that implements a subscriber interface
    • Subject emits events and allows observers to listen to the events
      • We can say attach observers to a subject
      • Or we can say an observer subcribes to a subject
  • https://refactoring.guru/design-patterns/observer/php/example#example-1
    • interface SplSubject
      • public function attach(SplObserver $observer): void;
      • public function detach(SplObserver $observer): void;
      • public function notify(): void;
        • Call SplObserver->update($this, $event, $data)
      • Initializing the observers in a SplSubject object
    • interface SplObserver
      • public function update(SplSubject $subject): void;
1.34.15.14. Chain of Responsibility (Behavioral DP) - Composition/Aggregation + Dependency Injection

1.34.16. Architectural Pattern

1.34.16.1. Model-View-Controller (MVC)
  1. Summary
    • Model can be included independently in other scenario while
      • Controller has to work with both the model and view
      • View
        • Notifies (not direct call but through event binding) Controller
        • Reads from Model directly
          • Not perfect
  2. Sample
    • main.py
      • model = Model()
      • view = TodoList(model)
      • controller = Controller(model, view)
      • controller.run()
    • view.py
      • bind_add_task(self, Callable[])
      • bind_delete_task(self, Callable[])
    • model.py
      • add_task
      • delete_task
      • get_tasks
    • controller.py
      • __init__
        • view.bind_add_task(self.add_task)
        • view.bind_delete_task(self.delete_task)
      • add_task
        • view.get_entry_text()
        • view.clear_entry()
        • model.add_task(task)
        • view.update_task_list()
      • delete_task
        • model.delete_task()
        • view.update_task_list()
1.34.16.2. Model-View-Presenter (MVP)
  1. Summary
    • Model
      • Notifies Presenter
      • Receives update requests from Presenter
    • Presenter
      • Receives notifications from Model and sends update requests to View
      • Receives notifications from View and sends update requests to Model
    • View
      • Receives update requests from Presenter and notifies Presenter
    • Compared to MVC
      • MVC
        • View couples with Controller and Model
        • Controller couples with Model and View
        • Model is independant
      • MVP
        • View couples with Presenter only
          • Presenter calls Model to get data and pass the data directly to View
          • Do other words, presenter is a controller in MVC where it does a bit more so that the View is not coupled with Model
        • Presentor couples with Model and View
        • Model is still independent
  2. Sample
1.34.16.3. Model–view–viewmodel (MVVM)
  1. Summary
    • Model
      • data centric
    • View Model
      • An abstraction of the view to handle data binding and event binding
    • View

1.34.17. Clean Code

  • Method names should have verbs
    • may choose to private constructor to force to use static factory methods to contruct a class
  • One level of abstraction per function
    • The Stepdown Rule
  • Organize switch into a factory method and appear only once
  • Function arguments
    • monadic, dyadic, triadic functions
    • Output argument
      • e.g. s is a StringBuffer object and let's output it
    • Instead of appendFooter(s);, s.appendFooter() is better
  • Prefer throwing exceptions to returning error codes
    • returning error codes
      • a lot of if statements
      • Enum is usually created and it is hard to maintain
    • catch exceptions in one place
    • One try/catch block, and place it at the beginning of the function that has it
  • One return in a function
  • Use less comment but better code (more functions, longer function names)
    • code is always up to date because it runs
    • comment might not be updated after code is changed
  • The Law of Demeter
    • A method f of a class C should only call the methods of these
      • C
      • An object created by f
      • An object passed as an argument to f
      • An object held in an instance variable of C
    • This violates the law
      • final String outputDir = ctxt.getOptions().getScratchDir().getAbsolutePath()
    • But this does not violate because it is just data structure
      • final String outputDir = ctxt.options.scratchDir.absolutePath
    • The best solution is:
      • final String outputDir= ctxt.createScratchFile()
    • Objects hide their data and expose operations (using interface)
  • Define the normal flow and use special case pattern to handle special cases
  • Create a wrapper of different types of exceptions (e.g. 3rd party API exceptions) to throw a new exception type
  • Don't return null, throw exception or return empty array
  • Don't pass null
  • Class Organization
    • public static constants
    • private static constants
    • private instance variables
    • rarely to have a public variable
    • public functions
    • private functions
  • a class or module should have one and only one, reason to change
    • Prefer more small classes
  • Cohesion
    • A class in which each variable is used by each method is maximally cohesive
    • The higher, the better
    • Functions that have multiple variables shared, group them into a class with high cohesion!
    • If another part of a large function does not share same variables as the first part, split them!
    • Types:
      • Functional cohesion
      • Sequential cohesion
      • Communicational cohesion
      • Procedural cohesion
      • Temporal cohesion
      • Logical cohesion
      • Coincidental cohesion
      • Procedural cohesion
      • Communicational cohesion
      • Temporal cohesion
      • Informational cohesion
      • Layer cohesion
  • Coupling
    • The degree of interdependence between software modules
    • The lower, the better
    • High coupling means that modules are closely connected and changes in one module may affect other modules.
    • Low coupling menas that modules are independent and changes in one module have little impact on other modules.
    • Types:
      • Data coupling
      • Stamp coupling
      • Control coupling
        • e.g. sort function that takes comparison function as an argument
      • External coupling
      • Common coupling
      • Content coupling
        • One module can modify the data of another module or control flow is passed from one module to the other module
        • Bad!
      • Temporal coupling
      • Sequential coupling
      • Communicational coupling
      • Functional coupling
      • Data-Structured coupling
      • Interaction coupling
      • Component coupling
  • Classes should be open for extension but closed for modification
    • Through adding new features in hierarchy

1.34.18. Namespace

  • Namespace names
    • case-insensitive
    • Don't declare namespace starting with backslash \
    • Don't use PHP as namespace name
  • __NAMESPACE__ magic constant to refer to the current namespace
  • Since PHP 5.3 supports most of the functionalities
  • Works for const, class, interfaces and function
  • Except declare statements, namespace should be the first line of the files
  • When code under a namespace refers to other components that have a different namespace or no namespace, need to refer to it using fully qualified name which starts with backslash \. e.g.
    • \Symfony\Component\HttpFoundation\Request
    • PHP built-in functions
  • When code under no namespace, can refer to other components without backslash
    • Symfony\Component\HttpFoundation\Request
    • new DateTime
  • Alias or import
    • FQDN
    • Alias are supported on PHP 5.3+
    • Aliases are supported on PHP 5.6+
    • use keyword must be declared in the outermost scope of a file or inside namespace declarations
      • The class being use~d can be defined later through ~include

    • e.g. 

      namespace DBH\Helpers;

use \Symfony\Component\HttpFoundation\Request;
// Or
// use \Symfony\Component\HttpFoundation\Request as SymfonyRequest;

// WebFramework is DBH\Helpers\WebFramework
class WebFramework extends Request {}
// Or
// class WebFramework extends SymfonyRequest {}

// Both are eq. to
// class WebFramework extends \Symfony\Component\HttpFoundation\Request {}

// import or alias multiple components from the same path/package since PHP 7
use \Symfony\Component\HttpFoundation\{Request, Response};

// Run a WebFramkework method inside this file with namespace DBH\Helpers
WebFramework::aStaticMethod();

./index.php 

include_once __DIR__."/../app/config.php";
include_once __DIR__."/../app/controllers/sys-curl.php"

./../app/config.php 

namespace MyApp\Abc;
function convert_to_kelvin() {}
const MYCONST = 1;
class CONFIG {
  static public function config() {
    return 'abc';
  }
}

./../app/controllers/sys-curl.php 

namespace MyApp\Abc;
class SYS_CURL {
  public function PULL() {
    $config = CONFIG::config();
  }
}

1.34.19. Cannot redeclare class

class_exists is case-insensitive match. Better to restart PHP first 

if (!class_exists('MyClass')) {
  class MyClass { }
}

1.34.20. ReflectionClass, get_class_vars(), get_object_vars()

1.34.20.1. Example
class A {

    const CONSTANT_1 = 123;

    public $publicInstance = 123;
    protected $protectedInstance = 123;
    /**
     * @var int|string
     */
    private $privateInstance = 123;
    public $publicInstanceNoDefault;
    private $privateInstanceNoDefault;
    public static $publicStaticNoDefault;
    public static $publicStatic = 123;
    private static $privateStaticNoDefault;
    private static $privateStatic = 123;

    public function getClassInfo() {
        var_export($test = get_class_vars(__CLASS__));
        echo PHP_EOL;
        var_dump(
            $test === array(
                'publicInstance' => 123,
                'protectedInstance' => 123,
                'privateInstance' => 123,
                'publicInstanceNoDefault' => NULL,
                'privateInstanceNoDefault' => NULL,
                'publicStaticNoDefault' => NULL,
                'publicStatic' => 123,
                'privateStaticNoDefault' => NULL,
                'privateStatic' => 123,
            )
        );

        var_export($test = get_object_vars($this));
        echo PHP_EOL;
        var_dump(
            'Get the accessible / visible non-static properties. Can be used outside of class',
            $test === array(
                'publicInstance' => 123,
                'protectedInstance' => 123,
                'privateInstance' => 123,
                'publicInstanceNoDefault' => NULL,
                'privateInstanceNoDefault' => NULL,
            )
        );

        $calledClass = get_called_class(); // vs __CLASS__
        $reflect = new ReflectionClass($calledClass);

        var_dump($test = $reflect->getConstants(), $test === array('CONSTANT_1' => 123));

        $filter = ReflectionProperty::IS_PRIVATE;
        $refectionProperties = $reflect->getProperties($filter);
        $privateProperties = array();
        foreach ($refectionProperties as $_property) {
            $docComment = $_property->getDocComment();
            preg_match('/@var (\S+)/m', $docComment, $typeFromDocComment);
            $typeFromDocComment = isset($typeFromDocComment[1]) ? $typeFromDocComment[1] : null;
            $typeFromDocComment = isset($typeFromDocComment) ? explode('|', $typeFromDocComment) : array();
            $propertyName = $_property->name;
            $className = __CLASS__;
            $propertyValue = ($_property->isStatic()) ? $className::$$propertyName : $this->$propertyName;
            /** @var string $propertyValueType
             * https://www.php.net/manual/en/function.gettype.php

             */
            $propertyValueType = gettype($propertyValue);
            $propertyValueTypeShortMap = array(
                'boolean' => 'bool',
                'integer' => 'int',
                // 'double' => 'float', // (for historical reasons "double" is returned in case of a float, and not simply "float")
                // 'string' => 'string',
                // 'array' => 'array',
                // 'object' => 'object',
                // 'resource' => 'resource',
                //  'resource (closed)' => 'resource (closed)', // as of PHP 7.2.0
                // 'NULL' => 'null',
                // 'unknown type' => 'unknown type'
            );
            $propertyValueTypeShort = isset($propertyValueTypeShortMap[$propertyValueType]) ? $propertyValueTypeShortMap[$propertyValueType] : $propertyValueType;
            array_push(
                $privateProperties,
                array(
                    'name' => $propertyName,
                    'class' => $_property->class,
                    'isDefault' => $_property->isDefault(), // whether it's declared at compile time
                    // 'isInitialized' => $_property->isInitialized(), // PHP 7.4.0+
                    'docComment' => $docComment,
                    'typesFromDocComment' => $typeFromDocComment, // array
                    'propertyValue' => $propertyValue,
                    'propertyValueType' => $propertyValueType,
                    'propertyValueTypeShort' => $propertyValueTypeShort,
                    'isTypeOneOfSpecifiedTypesFromDocComment' => in_array($propertyValueTypeShort, $typeFromDocComment),
                )
            );
        }
        var_export($privateProperties);
        echo PHP_EOL;
    }

}

$test = new A();
$test->getClassInfo();

1.34.21. Add capabilities to a class without making it bigger

https://gist.github.com/levonlee/5d26ba1df7d742df5d5c87be1fc7730d

1.34.22. Cast to object

  • If an object is converted to an object, it is not modified.
  • If a value of any other type is converted to an object, a new instance of the stdClass built-in class is created.
    • An array converts to an object with properties named by keys and corresponding values. Note that in this case before PHP 7.2.0 numeric keys have been inaccessible unless iterated.
  • If the value was null, the new instance will be empty.
$obj = (object) array('1' => 'foo');
var_dump(isset($obj->{'1'})); // outputs 'bool(true)' as of PHP 7.2.0; 'bool(false)' previously
var_dump(key($obj)); // outputs 'string(1) "1"' as of PHP 7.2.0; 'int(1)' previously

$a = new stdClass();
$a->{1} = 'abc';
$a->{'1'} = 'xyz';

var_dump($a->{'1'} === $a->{1});
var_dump($a->{1} === 'xyz');

1.35. Global

Define superglobal 

$GLOBALS['a'] = 'localhost';

// eq. to
// global $a;
// $a = 'localhost';

function body(){
    echo $GLOBALS['a'];
    // eq. to
    // global $a;
    // echo $a;
}

Pay attention to global scope 

<?php

// $a is not defined as a superglobal variable
$a = 'root a';
$b = 'root b';
global $c;
$c = 'root c';

global $d;
$d = 'root d';

function f1() {
    // $a is not defined as global inside a function
    // this causes sub level functions cannot access $a

    $a = 'a defined in f1';
    $b = 'b defined in f1';
    $c = 'c defined in f1';
    $d = 'd defined in f1';

    echo 'f1 a: ' . $a . PHP_EOL;
    echo 'f1 b: ' . $b . PHP_EOL;
    echo 'f1 c: ' . $c . PHP_EOL;
    echo 'f1 d: ' . $d . PHP_EOL;

    function f2() {
        global $a, $b, $c, $d;

        echo 'f2 a: ' . $a . PHP_EOL;

        $b = 'b defined in f2';
        echo 'f2 b: ' . $b . PHP_EOL;

        $c = 'c defined in f2';
        echo 'f2 c: ' . $c . PHP_EOL;

        echo 'f2 d: ' . $d . PHP_EOL;

        function f3() {
            global $a, $b, $c, $d;
            echo 'f3 a: ' . $a . PHP_EOL;
            echo 'f3 b: ' . $b . PHP_EOL;
            echo 'f3 c: ' . $c . PHP_EOL;
            echo 'f3 d: ' . $d . PHP_EOL;
        }

        echo 'running f3' . PHP_EOL;
        f3();

    }

    echo 'running f2' . PHP_EOL;
    f2();
}

f1();

/*
f1 a: a defined in f1
f1 b: b defined in f1
f1 c: c defined in f1
f1 d: d defined in f1
running f2
f2 a: root a
f2 b: b defined in f2
f2 c: c defined in f2
f2 d: root d
running f3
f3 a: root a
f3 b: b defined in f2
f3 c: c defined in f2
f3 d: root d
 */

1.36. Http Request - cURL, file_get_contents

1.36.1. cURL

<?php

$url = 'http://www.someurl.com'; // if this url is external, then it's safe to use.
// usually don't ever refer back to the same server because there might be firewall setting to prevent curl the local server

$myvars = 'myvar1=' . $myvar1 . '&myvar2=' . $myvar2;

// Or
$myvars = ['myvar1' => $myvar1, 'myvar2' => $myvar2];
$myvars = http_build_query($myvars); // default content-type is application/x-www-form-urlencoded

$ch = curl_init();
// use curl_setopt to set each one
// curl_setopt( $ch, CURLOPT_POST, 1);

// use curl_setopt_array to set multiple

// $verbose = fopen('php://temp', 'w+'); // enable debug

$options = array(
    CURLOPT_URL => $url,
    CURLOPT_POSTFIELDS => $myvars,
    // - If value is pure array, then request Content-Type will be set to multipart/form-data and Content-Length will be set (You don't need to specify)
    //     - CURLOPT_HTTPHEADER => array('Content-Type: multipart/form-data', 'Content-Length: 1234')
    // - If POSTFIELDS is a JSON string, set request Content-Type to application/json
    //     - CURLOPT_HTTPHEADER => array('Content-Type: application/json; charset=UTF-8', 'Content-Length: 1234')
    // - If POSTFIELDS is an XML string, set request Content-Type to XML
    //     - CURLOPT_HTTPHEADER => array('Content-Type: text/xml; charset=utf-8')
    // - If POSTFIELDS is a string and request Content-Type is not manually set, Content-Type is automatically set to x-www-form-urlencoded
    //     - use http_build_query($array) for POSTFIELDS
    CURLOPT_FOLLOWLOCATION => 1,
    // follow through. Default is not to follow through
    CURLOPT_HEADER => 0,
    CURLOPT_RETURNTRANSFER => 1,
    // CURLOPT_VERBOSE => true, // enable debug
    // CURLOPT_STDERR => $verbose, // enable debug
    // CURLOPT_USERAGENT => 'cPanel-Cron' // refer to waf:modsecurity, only works for GET request
);

// post json
$post = array('...');
$payload = json_encode($post);

$options = array(
    CURLOPT_URL => $url,
    CURLOPT_POST => 1,
    // post request with content-type: application/x-www-form-urlencoded header
    // For GET requests, just don't include CURLOPT_POST. Max 2048 bytes in URL restricted by browsers or servers. One TCP connection means the request sends http header and data together
    // For POST requests, browsers send http header, server responds with code 100 (continue), then browers send data (payload or CUROPT_POSTFIELDS), server responds with code 200
    CURLOPT_RETURNTRANSFER => 1,
    // return string rather than output
    CURLOPT_TIMEOUT => 5,
    // max. number of seconds
    CURLOPT_HTTPHEADER => array(
        'Content-Type: application/json',
        // one common request header that is needed
        'user-agent: curl/7.68.0',
    ),
    CURLOPT_POSTFIELDS => $payload
);

curl_setopt_array($ch, $options);

$response = curl_exec($ch);

// catch debug BEFORE close
/*
if (curl_errno($ch)) { var_dump(curl_errno($ch)); }
rewind($verbose);
$verboseLog = stream_get_contents($verbose);
echo "Verbose information:\n<pre>", htmlspecialchars($verboseLog), "</pre>\n";
*/

curl_close($ch);
1.36.1.1. Timeout
var_export(curl_version()['version']);
// Or shell run `curl --version`
// Since CURL v 7.60, CURLOPT_TIMEOUT_MS can be set

$ch = curl_init('https://a.com');
curl_setopt_array($ch, array(
    CURLOPT_CONNECTTIMEOUT => 2,
    CURLOPT_TIMEOUT => 4, // in seconds. For setting timeout less than 1 second, use CURLOPT_TIMEOUT_MS
    // CURLOPT_NOSIGNAL => 1, // If the timeout is set to less than a second using CURLOPT_TIMEOUT_MS, need to specify this to be 1 mainly because libcurl in multi-threaded needs it to make sure less than 1 second timeout is possible. Further info: libcurl timeout by default doesn't count domain name resolver time into timeout limit, use nosignal means ignore the domain name resolver signal (running) so that CURLOPT_TIMEOUT is the total time limit
    CURLOPT_RETURNTRANSFER => 1,
));
$response = curl_exec($ch)
$info = curl_getinfo($ch);
$errorNumber = curl_errno($ch);
$error = curl_error($ch);
// $info['http_code'] === 0, $errorNumber === 28, $error === 'Operation timed out after 15001 milliseconds with 0 bytes received' and $response === false if timeout
1.36.1.2. Manual multipart/form-data

Use CURLFile, requires physical files 

$someTextData = [
    'a' => 123,
    'b' => 'xyz',
];

$ch = curl_init("...");

$postData = [
    'data' => json_encode($someTextData),
    'file1' => new CurlFile('origin-name.pdf', 'application/pdf', 'download-name.pdf'),
    'file2' => new CurlFile('origin-name.pdf', 'application/pdf', 'download-name.pdf'),
];

curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postData);

$rs = curl_exec($ch);

Manually construct a multipart/form-data with binary file data 

<?php
$someTextData = [
    'a' => 123,
    'b' => 'xyz',
];

$ch = curl_init("...");

$postData = [
    'data' => json_encode($someTextData),
    'file1' => file_get_contents(__DIR__ . "/a.pdf"),
];

$postFiles = [
    'file1' => [
        'type' => 'application/pdf',
        'content' => file_get_contents(__DIR__ . "/a.pdf")],
];

$delimiter = '-------------' . uniqid();
$data = '';
//region Populate file fields
foreach ($postFiles as $name => $file) {
    $data .= "--" . $delimiter . "\r\n";
    // "filename" attribute is not essential; server-side scripts may use it
    $data .= 'Content-Disposition: form-data; name="' . $name . '"'
         . '; filename="' . $name . '.pdf"'
         . "\r\n"
         // this is, again, informative only; good practice to include though
         . 'Content-Type: ' . $file['type'] . "\r\n"
         // if data is binary, I found it unnecessary but include it here
         . 'Content-Transfer-Encoding: binary' . "\r\n"
         . '';
    // this endline must be here to indicate end of headers
    $data .= "\r\n";
    // the file itself (note: there's no encoding of any kind)
    $data .= $file['content'] . "\r\n";
}
//endregion

foreach ($postData as $name => $content) {
    $data .= '--' . $delimiter . "\r\n";
    $data .= 'Content-Disposition: form-data; name="' . $name . '"';
    $data .= "\r\n\r\n";
    $data .= $content . "\r\n";
}

// last delimiter
$data .= "--" . $delimiter . "--";

curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0); // default is 2. In current version, 1 means the same as 2. Use 2 instead of 1. 0 means the connection succeeds regardless of the names in the certificate. 2 means that it has to have the same name in the certificate as is in the URL you operate against
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0); // accept any server (peer) certificate. Default 1: enable CA checking. If this is 0, CURLOPT_SSL_VERIFYHOST will become 'does not matter'.
curl_setopt(
    $ch,
    CURLOPT_HTTPHEADER,
    [
        'Content-Type: multipart/form-data; boundary=' . $delimiter,
        'Content-Length: ' . strlen($data),
    ]
);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);

$rs = curl_exec($ch);
1.36.1.3. CURLOPT_ENCODING
  • Set the request header Accept-Encoding: and also enable decoding of the response
    • One single value
    • "", "identity", "deflate", and "gzip"
1.36.1.4. POST with CA
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_CAINFO, getcwd() . "/CAcerts/BuiltinObjectToken-EquifaxSecureCA.crt");
1.36.1.5. POST with SSL Certificate and Key

  • Convert PFX to a private key cl.pem and a certificate cl.cert. A passphrase is needed 

    openssl pkcs12 -in a.pfx -nocerts -out cl.pem -nodes
openssl pkcs12 -in a.pfx -nokeys -out cl.crt -nodes

# Test
curl -v --key ./cl.pem --cert ./cl.crt https://xxx

  • PHP 

    $curl = curl_init();
$xml = '';
curl_setopt($curl, CURLOPT_URL, "https://xxx.ca/endpoint?a=xyz");
curl_setopt($curl, CURLOPT_HTTPHEADER, array(
    'Content-Length: ' . strlen($xml),
    'Content-Type: application/xml',
));
$keyFile = 'cl.pem';
$certFile = 'cl.crt';
curl_setopt($curl, CURLOPT_POST, 1);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_POSTFIELDS, $xml);
curl_setopt($curl, CURLOPT_SSLKEY, $keyFile);
curl_setopt($curl, CURLOPT_SSLCERT, $certFile);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, true);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($curl, CURLOPT_VERBOSE, 1);
var_dump( curl_exec($curl));
var_dump( 'info', curl_getinfo($curl));
var_dump( 'error',  curl_error($curl));
var_dump( 'errno',  curl_errno($curl));
1.36.1.6. curl_error, http_code
$curl = curl_init();
$r['res'] = curl_exec($curl);
$r['info'] = curl_getinfo($curl);
$r['error'] = curl_errno($curl);
$r['error'] = $r['error'] ? "curl_errno: {$r['error']} " . curl_error($curl) : '';

if ($r['info']['http_code'] !== 200) {
    // 0 :: could be timeout
    echo $r['error'];
}
1.36.1.7. curl_getinfo()
// To see the request

$ch = curl_init('https://abc.com');
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
$res = curl_exec($ch);
echo "Request: " . PHP_EOL . curl_getinfo($ch, CURLINFO_HEADER_OUT) . PHP_EOL;

1.36.2. cURL-less

Use file_get_contents for PHP5. Some host might disable file_get_contents. Use cURL method Make sure php:ini:allow_url_fopen is enabled 

$url = 'http://server.com/path';
$data = array('key1' => 'value1', 'key2' => 'value2');

// use key 'http' even if you send the request to https://...
$options = array(
    'http' => array(
    'header'  => "Content-type: application/x-www-form-urlencoded\r\n",
    'method'  => 'POST',
    'content' => http_build_query($data)
    )
);
$context  = stream_context_create($options);
$result = file_get_contents($url, false, $context);

if ($result === FALSE) { /* Handle error */ }

// need to make sure string is UTF-8 php:string:encoding

var_dump($result);

1.37. HTTP Header

  • See header:cache-control 

    header("Cache-Control: no-store, no-cache, must-revalidate, max-age=0");
header("Cache-Control: post-check=0, pre-check=0", false);
header("Pragma: no-cache");

  • Set amount of time to cache 

    $seconds_to_cache = 3600;
$ts = gmdate("D, d M Y H:i:s", time() + $seconds_to_cache) . " GMT";
header("Expires: $ts");
header("Pragma: cache");
header("Cache-Control: max-age=$seconds_to_cache");

  • Array 

    header('response: ' . json_encode($response));

    var r = JSON.parse(rq.getResponseHeader('response'));
if (r.status) {
// ...
}
  • headers_list()
  • Return all HTTP request headers
    getallheaders()
    alias of apache_request_headers(). Only works for Apache web server
    • Earlier only works for PHP running in Apache as a module
    • Later supported for CGI and later from PHP 7.3 supported for FGM
    (no term)

    $_SERVER['HTTP_MY_REQUEST_HEADER']

    • Apache
    • If PHP is not running as a Apache module, then it's available
      • Any HTTP request header name will be converted to uppercase and dash to underscores

    • If PHP is running as an Apache module, do this 

      SetEnvIf MY_REQUEST_HEADER "(.*)" HTTP_MY_REQUEST_HEADER=$1

  • Don't echo before response header is set 

    // Don't do this
echo "MESSI is injured!!";
header("Location:somepage.php");

// Instead, do this
b_start();
echo "MESSI is injured!!";
header("Location:somepage.php");
ob_end_flush();

1.38. Throwable Interface

1.39. Exception 

try {
    throw new Exception('Some message', 30);
}
catch (Exception $e) {
    // final public Exception::getMessage(): string
    $e->getMessage();
    // final public Exception::getCode(): int
    $e->getCode();
}

1.40. Error Exception

  • https://www.php.net/manual/en/language.errors.php7.php
    Parent class
    see 1.38
    (no term)
    Use catch (Error $e) {...}, or catch (Throwable $t) {...} or set_exception_handler to handle fatal error
    • If an Error exception is not try catch and set_exception_handler is not set, then Error exception becomes a fatal error, and later set_error_handler deals with the fatal error
    • PHP 7 now throws Error exceptions
    (no term)
    Child classes
    • ArithmeticError
    • AssertionError
    • CompileError
      • ParseError
    • TypeError
      • ArgumentCountError
    • ValueError
    • UnhandledMatchError
    • FiberError
try {
    // throw new Exception('an exception');
    // throwAFatalError();
    // die does not throw fatal error nor exception
    die('throw a fatal error');
}
catch (Exception $e) {
    echo "This is an exception: " . $e->getMessage();
}
catch (Error $e) {
    echo "This is a fatal error: " . $e->getMessage();
}

1.41. Type declaration

  • https://www.php.net/manual/en/language.types.declarations.php
  • Can be added to
    • function arguments
    • return values
    • class properties since PHP 7.4
  • A child's method must match any return type declartion of the parent
    • Parent may not define a return type, then the child method may do so
  • Types
    void
    since PHP 7.1
    nullable type
    since PHP 7.1
    ?string
    either a string or null
    function f (C $c = null) {}
    this is also possible but not recommended
    self, parent
    There's no static in PHP 7.x! An instance of the same class
    static
    since PHP 8.0
    mixed
    since PHP 8.0
    Union types
    since PHP 8.0
    • string|null|int
    Intersection typen
    since PHP 8.1
    (no term)
    Some others

1.42. Pass by Reference

  • Don't easily pass by reference in foreach loop 

    $a = array ('zero','one','two', 'three');

foreach ($a as &$v) {

}

// $v is a reference of $a[3]

foreach ($a as $v) {
  // $a[3] is changed to zero, one and two
  echo $v . PHP_EOL;
}

// zero one two two

1.43. Troubleshooting

1.43.1. Parse error: syntax error, unexpected end of file

Or Parse error: syntax error, unexpected 'endwhile' (t_endwhile)

Check if php short_open_tag is enabled. Default is not enabled. Then search <? ~ with a space and replace them with ~<?php

1.44. Naming Convention

  • PSR Naming Conventions
  • OOP
    • ClassNamesLike
    • MyTrait
    • methodName
    • propertyName
    • const CONSTANTS_LIKE_THIS
  • global functions
  • local functions
  • global variables
    • Global names must be prefixed or use namespace
  • $localVariableName
  • define('CONSTANTS_LIKE_THIS', '');
    • - Global names must be prefixed or use namespace
  • File and directory

    • No standard but I prefer

      Directory
      public_html

      • I would avoid using hypen and use nested directory structure

        • lawyer/lawyerone lawyer/laywertwo
        File
        myclassname_is_long.class.php

1.45. Shell Exec Command

echo shell_exec("whoami");

1.46. Session

  • PHP relies on client cookie PHPSESSID to identify session
  • js:xmlhttprequest made on the same domain always transfer all cookies
  • php:ini:session

  • Need to start session. check if session is already started 

    if (session_id() == "")
  session_start();

1.47. Predefined Interfaces and Classes

1.47.1. IteratorAggregate and Iterator (interfaces) 

interface IteratorAggregate extends Traversable {

    /* Methods */
    public getIterator(): Traversable
}

interface Iterator extends Traversable {

    /* Methods */
    public current(): mixed
    public key(): mixed
    public next(): void
    public rewind(): void
    public valid(): bool
}

1.48. Composer

1.48.1. Install

1.48.1.1. Ubuntu
# php8.2-curl is required
# apt install php8.2-curl

# It's better to install zip/unzip (or 7z) rather than relying on php8.2-zip
# apt install zip unzip

# download installer, run it using PHP and put the resulted executable as in /usr/local/bin/composer
curl -sS -L https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer

composer --version

1.48.2. Basics

composer require abc/xyz
downloads the latest version to current folder ./vendor/composer/abc/xyz/*.* and add to require in composer.json
composer require abc/xyz:0.3.*
specify a range of versions
To use this package in PHP file
require 'vendor/autoload.php'; use Abc\Xyz\component1;
(no term)
Whether packages are installed locally or globally, it will use the user's directory ~/.composer. If current user is not root, don't run composer with sudo
(no term)
composer global require "abc/xyz"
  • global package is installed
  • Installed path default is ~/.composer/vendor/bin
    • composer global config bin-dir --absolute to find out the exact path
    • make sure the path is in PATH
      • echo 'export PATH="$PATH:/root/.config/composer/vendor/bin"' >> ~/.bashrc
      • source ~/.bashrc
(no term)
composer global remove abc/xyz
(no term)
composer.json
composer init
initialize proj and create composer.json
composer install
install packages based on composer.json
(no term)
require
  • "abc/packagename": "0.3.*"
  • ">~1.2 eq. to >=1.2 <2.0.0, ~1.2.3 is eq. to >=1.2.3 <1.3.0
  • ^1.2.3 is eq. to >=1.2.3 < 2.0.0
(no term)
require-dev
  • added by composer require --dev
(no term)
composer outdated
(no term)
composer update or composer update abc/pkgname
composer show
show installed packages
(no term)
composer remove [vendor/package]
(no term)
composer self-update
  • composer -V
.gitignore
/vendor/

1.48.3. composer.json

autoload
after edit, run composer dump-autoload -o

  • psr-4 

    "autoload": {
"psr-4": {
    "TDD\\": "src/"
}
},
    • Map a namespace TDD to directory src
    • "psr-4": { "": "src/" }

  • build map by scanning for classes in all .php and .inc files in specified directories/files 

    {
"autoload": {
    "classmap": ["src/", "lib/", "Something.php"]
}
}

1.48.4. Packages

PHP Package Repo
https://packagist.org/
1.48.4.1. PHP Code Sniffer
1.48.4.2. squizlabs/php_codesniffer
composer global require "squizlabs/php_codesniffer=*"

# Make sure composer bin dir in PATH
# Check the composer bin dir location
# default should be ~/.composer/vendor/bin/
# - 2 files are installed under this directory: phpcs and phpcbf
# - ~/.composer/vendor/squizlabs/php_codesniffer/src/Standards/PSR12 is the installation path for standard PSR12
composer global config bin-dir --absolute

# export PATH="$PATH:$HOME/.composer/vendor/bin"

# My docker container is /root/.config/composer/vendor/bin
echo 'export PATH="$PATH:/root/.config/composer/vendor/bin"' >> ~/.bashrc
source ~/.bashrc

phpcs --version

# see all installed coding standards
# PSR12 is installed by default
phpcs -i

# List all sniffers included in a standard
phpcs --standard=PSR12 -e

cd /path/to/my-project

phpcs --standard=PSR12 sample.php
1.48.4.3. phpcodesniffer-composer-installer
# https://github.com/PHPCSStandards/composer-installer
# It can install phpcs and also standards

composer require --dev dealerdirect/phpcodesniffer-composer-installer

  • Sample 

    {
    "require-dev": {
        "dealerdirect/phpcodesniffer-composer-installer": "^0.4",
        "wp-coding-standards/wpcs": "^2",
        "phpcompatibility/php-compatibility": "^9"
    },
}
  • And run composer install
  • PHP_CodeSniffer is installed at ./vendor/bin/phpcs and it links to multiple standards installed above

1.49. PHPUnit

  • https://phpunit.de/getting-started/phpunit-9.html
    v9
    PHP 7.3, 7.4
    v8
    PHP 7.2, 7.3
  • composer require --dev phpunit/phpunit ^9
    • ./vendor/bin/phpunit --version

  • composer.json should look like 

    {
"autoload": {
    "classmap": [
    "src/"
    ]
},
"require-dev": {
    "phpunit/phpunit": "^9"
}
}
  • Code to test src/Email.php, Test file is tests/EmailTest.php. A test is a public method named test* or add @test annotation in a method's docblock
  • ./vendor/bin/phpunit tests
    Run one test file
    ./vendor/bin/phpunit tests/ReceiptTest.php
    Filter test cases in files and run tests

    ./vendor/bin/phpunit tests --filter=testTax

    Run function that matches the regex testTax
    e.g. matched methodName
    (no term)
    ./vendor/bin/phpunit tests --filter=ReceiptTest::testTax
    • Old version of PHPUnit
      • phpunit --filter testTax path/to/ReceiptTest.php
    Filter by testsuite
    ./vendor/bin/phpunit --testsuite=app --filter=testTax
  • https://phpunit.readthedocs.io/en/8.5/assertions.html
    • assertEquals(mixed $expected, mixed $actual[, string $message = ''])
  • phpstorm:phpunit

1.49.1. phpunit.xml

  • Set filter, testsuites (groups), colors
    • <testsuites>
      • <testsuite>
    • logging

1.49.2. CLI

  • Syntax
    • phpunit --help
    • phpunit [options] UnitTest.php
    • phpunit [options] <directory>
  • Characters to indicate progress
    .
    success
    F
    fail
    E
    error
    R
    marked as risky
    S
    skipped
    I
    incomplete or not yet implemented
  • Options of config
    -d key[=value]
    set a php.ini value e.g. -d memory_limit=256M
    --configuration, -c
    e.g. -c sub/phpunit.xml read config. If omitted, phpunit.xml or phpunit.xml.dist at current dir. will be read
  • Options of test selection
    --testsuite <name,pattern,...>
    which testsuites to run
  • Options of Test Execution
    • --debug

1.50. Standard PHP Library (SPL)

1.50.1. Interfaces

1.50.1.1. SplObserver
1.50.1.2. SplSubject

1.51. Libraries

1.51.1. mPDF

Convert HTML to PDF https://github.com/mpdf/mpdf https://mpdf.github.io/css-stylesheets/supported-css.html

1.51.1.1. @page

https://mpdf.github.io/paging/using-page.html 

@page {
  margin-top: 1cm;
  margin-bottom: 3cm;
  margin-left: 2cm;
  margin-right: 2cm;
}

body {
     background-image: url(paper-size-ratio.jpg);
     background-repeat:no-repeat;
     background-position:top center;
     background-image-resize:6;
}

Paper size

A4
210 × 297 millimeters or 8.27 × 11.69 inches

1.51.2. Mobile-Detect

Device tect, browser detect :: GitHub Examples 

require_once 'Mobile_Detect.php';
$detect = new Mobile_Detect;

$detect->isIphone();
$detect->isSamsung();
$detect->is('iphone');
$detect->version('Android');

if ($detect->isMobile()) {}

if ($detect->isTablet()) {}

// Check for any mobile device, excluding tablets.
if ($detect->isMobile() && !$detect->isTablet()) {}

1.51.3. firebase/php-jwt

https://github.com/firebase/php-jwt

1.51.4. Laravel

1.51.4.1. Installation
A skeleton Laravel project runs on top of the Laravel Framework
https://github.com/laravel/laravel
  • composer create-project laravel/laravel example-app
Laravel Framework
https://github.com/laravel/framework
(no term)
chmod -R 777 storage
  1. Nginx
    server {
        server_name my.website.com;
        root /var/www/mywebsite/public;

        add_header X-Frame-Options "SAMEORIGIN";
        add_header X-Content-Type-Options "nosniff";

        index index.php;

        charset utf-8;

        location / {
                try_files $uri $uri/ /index.php?$query_string;
        }

        location = /favicon.ico { access_log off; log_not_found off; }
        location = /robots.txt  { access_log off; log_not_found off; }

        error_page 404 /index.php;

        location ~ \.php$ {
                fastcgi_pass unix:/var/run/php/php8.2-fpm.sock;
                fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
                include fastcgi_params;
        }

        location ~ /\.(?!well-known).* {
                deny all;
        }

        listen 80;
}
1.51.4.2. Sail
  1. Installation (Laravel Framework 9.9.0)
    • Change APP_PORT in 
    curl -s https://laravel.build/example-app | bash

cd example-app
# Interactive mode for debugging
./vendor/bin/sail up

# Detached mode
./vendor/bin/sail up -d

# stop all running containers
./vendor/bin/sail stop

# Get status of containers (remove exited containers)
./vendor/bin/sail status

# nano ./.env
# Add
APP_PORT=81 # To solve Bind for 0.0.0.0:80 failed: port is already allocated

# Then go to http://localhost:81

# Laravel Source Code
cd ./vendor/laravel/framework/src/

# e.g. ORM
cd Illuminate/Database/Eloquent/
  2. CLI
    # Display Help
./vendor/bin/sail

# Container CLI
sail shell
# eq.
sail bash

sail root-shell
sail root-bash

# Artisan commands
sail artisan --version
  3. .env
1.51.4.3. Artisan CLI
php artisan --version
# List of all commands
php artisan list

cd /path/to/my/website

# manually modify .env file

# whenever .env file is created or updated
php artisan config:cache

# generate key, .env > APP_KEY is modified
php artisan key:generate

php artisan config:cache

cd database/seeders

php artisan migrate

# .env > JWT_SECRET is inserted
php artisan jwt:secret

php artisan config:cache
1.51.4.4. Eloquent (ORM)
  1. Model.php
    1. Traits
      • Concerns\HasRelationships
  2. Relationships

    https://laravel.com/docs/9.x/eloquent-relationships

    1. Local scope
1.51.4.5. Architecture
  1. Short Version
    • create an app
      • attach singleton of kernals: like api, http kernels
    • Instantiate the kernal
    • Ask kernal to handle a request
      • Kernal runs register for each service provider defined in config/app.php
        • Attach/bind more things to the $app (dependency injection)
          • Instances of Service Container
          • Facade, which can be:
            • singleton of a service container
            • singleton of any class!

        • To access $request in Service Provider's register has no access $request 

          // Contextual Binding
public function register() {
    $this->app->when(Service::class)
        ->needs(Thingable::class)
        ->give(function () {
            return request()->user_name === 'userA'
                ? $this->app->get(DependencyA::class)
                : $this->app->get(DependencyB::class);
        });
}

// or

public function register() {
    $this->app->bind('App\Thingable', function ($app) {
        return $app->make(request()->user_name === 'userA' ? 'App\DependencyA' : 'App\DependencyB');
    });
}
      • Kernal runs boot for each service provider
      • Kernal runs all middlewares which are defined when service providers are registered
      • Kernal deals with the request
        • Pick a route that is defined at routes/web.php
          • Instantiates the corresponding controller defined in the route
          • And do the method of the controller e.g. show loan, delete loan
            • Either use the previously injected dependencies (from the $app)
            • or use the Facade
      • Kernal runs all middlewares which are defined when service providers are registered, run them before delivering the response
      • Kernal delivers the response
    • Return the response after dealing a request
    • Terminate the kernel
  2. Long Version
    • public/index.php
      • bootstrap/app.php
        • create an instance of the application (service container which extends Illuminate\Container\Container)
          • Request is sent to either the HTTP kernel (app/Http/Kernel.php) or the console kernal.
            • HTTP kernel's handle method is simple: receive Request and return Response
              • Think of the kernel as being a big black box that represents the entire app.
            • HTTP kernel extends Illuminate\Foundation\Http\Kernel
            • bootstrappers are defined and run before the request is executed
              • config error handling
              • config logging
              • detect app environment
              • perform other tasks that need to be done before the request is handled
              • One of the most important kernel bootstrapping actions is loading the Service Providers
                • Service providers are configured in configure/app.php, the providers array
                  • Each provider is instantiated and the corresponding methods are called:
                    • register method is called
                    • boot method
                • Routing is a service provider
                  • App\Providers\RouteServiceProvider loads routes directory
                  • Once the app is bootstrapped and all service providers are registered, Request will be handled off to the router for dispatching. The router will dispatch the request to a route or controller, as well as run any route specific middleware.
                  • After all the route's assigned middleware has passed through, the route or controller method will be executed and the response returned by the route or controller method will be sent back through the route's chain of middlewares
                  • Once the route or controller method returns a response, the response will travel back outward through the route's middleware, giving the app a chance to modify or examine the outgoing response.
                  • Finally, once the response travels back through the middleware, the HTTP kernel's handle method returns the response object and the index.php calls the send method on the returned resposne. The send method sends the response content to the user's web browser.
            • Middlewares are defined
              • all requests must pass through before being handled by the application
                • read and write HTTP session
                • determine if the app is in maintenance mode
                • verify CSRF token and more
                • determine if the user is logged in

1.52. Run PHP Online in multiple versions

1.53. Coding Standard

1.53.1. Variable, action/filter and function name

  • all lowercase, never camelCase
  • Separate words by underscore

1.53.2. Class name and file name

  • Class name
    • WordPress
      • Capitalized words separated by underscores
      • Acronym should be all upper case
    • General PHP
      UpperCamel
      can have underscores and numbers
      (no term)
      SampleXmlClass, not SampleXMLClass
      (no term)
      Names should not include Drupal, Class
      (no term)
      Interface should always prefix Interface
  • Class file name
    • lowercase with class- preppended
    • WP_Error is class-wp-error.php
1.53.2.2. Aggregation vs Composition
Aggregation
object A contains objects B; B can live without A.
Composition
object A consists of objects B; A manages life cycle of B; B can’t live without A.

2. Drupal

2.1. Install Drupal on Windows

https://www.drupal.org/node/970492

2.2. Update Core

  • pantheon:drupal:core

  • For Windows, you may need to stop and do the file changes and then start again. Restart IIS 

    iisreset -stop
iisreset -start

# this stops and starts
iisreset
  • https://www.drupal.org/docs/7/updating-your-drupal-site/how-to-update-drupal-core
  • Summary
    • Make backup
    • Download and Extract
    • Set website on maintenance mode
    • Delete all files and folders
      • don't delete folder /sites
      • delete folders /profiles/minimal /profiles/standard /profiles/testing
      • delete folders /includes /misc /modules /scripts /themes
      • reapply .htaccess robots.txt web.config settings.php
      • reapply any modified files and folders
    • Some updates don't include settings.php. If it includes, replace the old one in sites/default with the new one and apply your previous changes
    • replace favicon.ico
    • Login as user 1 and run update.php
    • If you are unable to access update.php do the following: (allow update.php to be run when not logged in as admin)
      • Open settings.php with a text editor.
      • Find the line that says: $update_free_access = FALSE; Change it into: $update_free_access = TRUE;
      • Try again to run update.php.
      • Once the update is done, $update_free_access must be reverted to FALSE.
      • In a multi-site installation, run update.php again for each site.
    • Navigate to Administration > Configuration > Development > Performance and clear all cache
    • Admin > Reports > Status to Verify everything is working as expected.
    • Ensure that $update_free_access is FALSE in settings.php.
    • Disable maintenance mode

  • https://www.drupal.org/project/module_missing_message_fixer to overcome these errors 

    The following module is missing from the file system: MODULE NAME. In order to fix this, put the module back in its original location. For more information, see the documentation page.

    Or 

    User warning: The following module is missing from the file system: MODULE NAME. In order to fix this, put the module back in its original location. For more information, see the the documentation page. in _drupal_trigger_error_with_delayed_logging()
  • In order to check available updates for core and modules, module Update Manager needs to be enabled

2.2.1. 7.x

Legend

  • If no changes for .htaccess, web.config, robots.txt nor settings.php, then there's no description. Otherwise, it will indicate which ones to update.

7.50 :: d7:ts:module is missing robots.txt, default.settings.php, add .editorconfig file 7.51, 7.52 7.53 :: update jQuery to 1.7-1.11.0 7.54 7.55 7.56 :: security 7.57 :: security 7.58 :: security

2.2.2. Troubleshoot

2.2.2.1. The following module is missing from the file system d7:ts:module is missing

Because the module code no longer exists, so I need to manually remove traces in db. D7 

drush sql-query "DELETE from system where name = 'old_module1' AND type = 'module';"

If module name is not found in db, most likely the code incorrectly refers to a non-existing module such as drupal_get_path('module', 'deletedmodname'). Do backtrace to identify.

2.3. Update Module

  • Read module README first
  • Usually just remove the module directory and put the new one in
  • Run update.php

2.4. Module

2.4.1. Core Modules

2.4.1.1. Path d7:proj:path
  • Set the path for an individual node with the Path module (on the node/add or node edit form)
  • Add an URL alias at: Administer > Configuration > Search and metadata > URL aliases
  • Administer the list of URL aliases at: Administer > Configuration > Search and metadata
  • Administer > Configuration > Clean URLs
  • https://www.drupal.org/docs/7/core/modules/path/overview
  • May use d7:proj:pathauto
2.4.1.2. Entity API d7:entity
Sub module
entity_token
(no term)
Requires nothing
(no term)
Update
  • 1.5
  • 1.9 no db

2.4.2. Custom Module

2.4.2.1. .info file
  • Instructions
  • Module name must contain only lower-case and undersocre
    • sites/all/modules/custom/my_module
    • sites/all/themes/custom/my_theme
    • my_module.info
    • my_module.module
  • Use ; at the beginning of a line to mark as comment
  • name = A Human Readable Name
  • description = More is <a href="">here</a>. Only one line with 255 characters and it's HTML (e.g. accented characters)!
  • core = 7.x Can't specify minor version
  • package = Views package = Example modules
    • If your module comes with other modules or it's meant to be used exclusively with other modules, provide the package name here.
    • More than 4 modules that depend on each other can make a package.
  • files[] = includes/context.inc This .inc file should be class or interface. To include .inc with plain functions
  • Clear cache when .info is changed
  • A submodule under a package may not add parent module to dependencies and submodule names can be anything given the parent module is examples:
    • action_example
    • action_examples
    • example_action
dependencies[] = entity
dependencies[] = views (>=3.12)
dependencies[] = exampleapi (1.x)
dependencies[] = exampleapi (>7.x-1.5)
dependencies[] = exampleapi (>1.0, <=3.2, !=3.0)
dependencies[] = system (>=7.53)

Optional 

configure = admin/config/content/my_mymodule

; hide on modules page. e.g. SimpleTest where end-users should never enable the testing modules
hidden = TRUE
2.4.2.2. .install File
  • Define new tables, load data and implement conversions during updates
  • It's run the first tiem a module is enabled and is used to run setup procedures as required by the module
  • No syntax. It's a PHP file
  • https://www.drupal.org/docs/7/creating-custom-modules/writing-install-files-drupal-7x

  • called when the module is first enabled. Typical use is to create the necessary tables d7:hook_install 

    function your_module_name_install() {
    // change module weight
    db_update('system')
        ->fields(array('weight' => 999)
        ->condition('name', 'your_module_name', '=')
        ->execute();

    // 1 heavier than another module's weight
    // Get the weight of the module we want to compare against
    $result = db_select('system', 's')
        ->fields('s', array('weight'))
        ->condition('name', 'the_other_module_name', '=')
        ->execute();
    $weight = !empty($result) ? $result->fetchField() : 0;

    // Set our module to a weight 1 heavier, so ours moves lower in execution order
    db_update('system')
        ->fields(array('weight' => $weight + 1))
        ->condition('name', 'your_module_name', '=')
        ->execute();
}
  • hook_schema()
  • will be called by update.php
    • 1 digit for Drupal core compatibility
    • 1 digit for module's major release version. Use 0 for initial porting of the module to a new Drupal core API
    • 2 digits for sequential counting, starting 00
    • The required update for mymodule to run with Drupal core API 7.x when upgrading from 6.x
    • The first update to get the db ready to run mymodule 7.x-1.*
    • The first update to get the db ready to run mymodule 7.x-2.*. Users can directly update from 6.x-2.* to 7.x-2.* and they get all 70xx and 72xx updates, but not 71xx updates, because those reside in the 7.x-1.x branch only
2.4.2.3. Refactor, include .inc files
function ctools_include($file, $module = 'ctools', $dir = 'includes') {
  static $used = array();

  $dir = '/' . ($dir ? $dir . '/' : '');

  if (!isset($used[$module][$dir][$file])) {
    require_once DRUPAL_ROOT . '/' . drupal_get_path('module', $module) . "$dir$file.inc";
    $used[$module][$dir][$file] = TRUE;
  }
}

// ctools_include('utility', 'ctools', 'includes');
// ctools_include('macro', 'mymodule', 'includes');
2.4.2.4. lili.api.php

Just include this in module root directory and all hooks will be defined

2.4.3. sqlsrv

Install PDO Microsoft Drivers for PHP for SQL Server https://docs.microsoft.com/en-us/sql/connect/php/system-requirements-for-the-php-sql-driver?view=sql-server-2017

For PHP 5.x, use PDO 3.2 For PHP 7.x, use PDO 5.2

PDO 4.x and 5.x supports PHP 7 only. https://docs.microsoft.com/en-us/sql/connect/php/system-requirements-for-the-php-sql-driver?view=sql-server-2017#driver-versions

Download 4.0, 3.2, 3.1, 3.0 https://www.microsoft.com/en-us/download/details.aspx?id=20098 run SQLSRV32.EXE on the real server, select the version and any place to extract the drivers.

To download other versions for PHP 7 :: https://docs.microsoft.com/en-us/sql/connect/php/download-drivers-php-sql-server?view=sql-server-2017

To find out which PHP version is used in IIS :: Your website > Handler Mappings > FastCgiModule Setup PHP on IIS https://docs.microsoft.com/en-us/iis/web-hosting/web-server-for-shared-hosting/fastcgi-with-php

If php5.dll is used, then it's non-threaded, if php5ts.dll (thread safe), then it is threaded. Use the correct version of the SQLSERV and PDO_SQLSERV drivers.

Non-thread safe :: php_sqlsrv_54_nts.dll php_pdo_sqlsrv_54_nts.dll

Put those into ext folder. Check extension_dir in php.ini.

Change these lines in php.ini 

;extension=php_pdo_sqlsrv.dll
;extension=php_sqlsrv.dll

extension=php_pdo_sqlsrv_54_nts.dll
extension=php_sqlsrv_54_nts.dll

Run iisreset to restart IIS.

2.4.4. Pathauto d7:proj:pathauto

  • The Pathauto module automatically generates URL/path aliases for various kinds of content (nodes, taxonomy terms, users) without requiring the user to manually specify the path alias. This allows you to have URL aliases like /category/my-node-title instead of /node/123. The aliases are based upon a "pattern" system that uses tokens which the administrator can change
  • Requires

2.4.5. Feeds - Package: Feeds

Requires d7:ctools d7:job_scheduler Test requires: date:date entity_translation:entity_translation feeds_xpathparser:feeds_xpathparser i18n:i18n_taxonomy link:link rules:rules variable:variable Submodules:

  • feeds_import
  • feeds_news
  • feeds_ui
$feed_obj = feeds_source($feed_id,$feed->feed_nid);
$feed_obj->existing()->import(); // Use this so no jobs are inserted in batch

// Sequence
$this = feeds_source;
hook_feeds_before_import($this)
hook_feeds_after_parse($this, $parser_result)
FeedsProcessor.inc->process parser_result
releaseLock()
hook_feeds_after_import($this)

Import drush command runtime can be long. Output something every 5 minutes to prevent drush command timeout Do this in one of the hooks:

  • hook_feeds_before_update
  • hook_feeds_presave
  • hook_feeds_after_save
function lili_feeds_presave(FeedsSource $source, $entity, $item) {

 $request_time = lili_custom_cache('timecounter');
 $interval = 60 * 5;
 if (!is_null($request_time)) {
  $row_counter = lili_custom_cache('rowcounter');
  lili_custom_cache('rowcounter',++$row_counter);
  $current_time = time();
  $seconds = $current_time - $request_time;
  if (floor($seconds/$interval) > 0) {
   lili_custom_cache('timecounter', $current_time);
   print "Presave row #$row_counter. $seconds seconds have passed.\n";
  } 
 }
 else {
  lili_custom_cache('timecounter', REQUEST_TIME);
  lili_custom_cache('rowcounter', 1);
 }

}
2.4.5.1. New Feed Tamper Plugin

The plugin does this: Drupal download file with real file extension 

// Implements hook_ctools_plugin_directory()
function lili_ctools_plugin_directory($module, $plugin) {
 if ($module == 'feeds_tamper') {
  return 'plugins'; // path_to_lili_module/plugins/lili_*.inc
 }
}

// lili_change_image_extension.inc
// You can refer to path_to_feeds_tamper_module/plugins/explode.inc
$plugin = array(
  'form' => 'lili_image_extension_form',
  'callback' => 'lili_image_extension_callback',
  'validate' => 'lili_image_extension_validate', // optional
  'name' => 'Convert Image URL Extension',
  'category' => 'T and T', // group name
  'multi' => 'direct',
  // If multiple values are injected to the plugin,
  // 'direct' will pass the whole array as $field to callback
  // while 'loop' will loop over and pass each value to callback
);

function lili_image_extension_form($importer, $element_key, $settings) {
  $form = array();
  $form['info'] = array(
    '#markup' => t('Converts source image URLs that don\'t have regular image extension: jpg, png, etc.'),
  );

  // Extra variables to pass to $settings in callback
  $form['importer_name'] = array(
  '#type' => 'hidden',
  '#title' => t('Importer machine name'),
  '#default_value' => isset($settings['importer_name']) ? $settings['importer_name'] : $importer->id,
  );

  return $form;
}

function lili_image_extension_validate(&$settings) {
  // Validate $settings.
}

function lili_image_extension_callback($result, $item_key, $element_key, &$field, $settings, $source) {
  //settings has the importer name
  //$settings['importer_name']
  //$field has the importing field text value
  if (!is_array($field)) {
    $field = array($field);
  }

  $val = array();

  foreach ($field as $f) {
    $val[] = _lili_save_file_with_extension($f);
  }
  $field = $val;
}
2.4.5.2. XPath

Children nodes Photo under Images node. Simply Images/Photo, the result is already array Concate multiple nodes concat(photo1,';',photo2) then explode

2.4.6. Feeds Tamper - feeds_tamper - Package: Feeds

Requires feeds Submodule: feeds_tamper_ui

2.4.7. Feeds Tamper Importer - feeds_tamper_importer - Package: Feeds

Requires feeds_tamper

2.4.8. Feeds XPath Parser - feeds_xpathparser - Package: Feeds

Requires feeds

2.4.9. Feeds entity processor - feeds_entity_processor - Package: Feeds

Requires feeds d7:entity

2.4.10. XML Sitemap

admin/config/search/xmlsitemap/settings Uncheck /Prefetch URL aliases during sitemap generation

2.4.11. ThemeKey d7:module:themekey

You can switch to a different based on some rules! If you are not clear about some rules, you can just search in code. e.g. search 'system:query_string'

2.4.12. Devel

  • requires nothing
  • devel_generate, devel_node_access (off)

http://ratatosk.net/drupal/tutorials/debugging-drupal.html

dpm($input, $name = null) use drupal_set_message and Krumo to dispaly in 'message' area kpr($input, $return = FALSE, $name=NULL) uses Krumo print in page header not 'message' area. dvm($input, $name = NULL) use Krumo to var_dump in 'message' area. Good for copy and paste dpr($input, $return = FALSE, $name = NULL) print in page header not in 'message' area.

2.4.13. Administrator menu - admin_menu - Package: Administration

Requires nothing Submodules: admin_devel, admin_menu_toolbar

2.4.14. Administration Views - admin_views - Package: Administration

Requies d7:views d7:views_bulk_operations

2.4.15. Administer Users by Role - administerusersbyrole

Requires chain_menu_access

Allow users to edit/delete other users

2.4.16. Variable

variable requires nothing d7:m:variable

2.4.17. Libraries

requires nothing

2.4.18. Module Filter

module_filter :: requires nothing

2.4.19. Geocoder

Requires d7:geophp d7:ctools d7:entity d7:geocoder An API and widget to geocode text field into GIS data types.

2.4.20. Entity View Modes - entity_view_mode

Requires nothing d7:entity_view_mode Recommends d7:field_ui Create custom View Mode: Default, Teaser under Manage Display d7:ds can also create view mode

2.4.21. wysiwyg

requires nothing Allows the use of client-side editors to edit content 7.x-2.2 to 7.x-2.4 (support TinyMCE from 3.3.9.2 to 4.5.7)

2.4.21.1. Installation & Config

Setup assign text format to trusted roles: admin/config/content/formats Follow README.md to config Text Format (e.g. Full HTML). Basically remove any restrictions. Installation Instructions on config page: /admin/config/content/wysiwyg TinyMCE installation instructions are missing.. Refer to this https://www.drupal.org/docs/7/modules/wysiwyg/installation Basically, download from TinyMCE and put it in sites/all/libraries/tinymce

If TinyMCE editor is upgraded, make sure the options for the Wysiwyg profile using TinyMCE is the same as before admin/config/content/wysiwyg/profile e.g. https://www.todaystrucking.com/admin/config/content/wysiwyg/profile/full_html/edit You need to save the profile after the TinyMCE editor is updated.

Supported editor versions: https://www.drupal.org/project/wysiwyg

  1. All TinyMCE settings

    To config other TinyMCE settings and the Wysiwyg module UI doesn't provide, use hook_wysiwyg_editor_settings_alter()

    1. HTML elements are stripped off

      In /admin/config/content/wysiwyg/profile/full_html/edit, keep Verify HTML on. 

      function lili_wysiwyg_editor_settings_alter( &$settings, $context ) {
  dpm($settings,  'settings');
  dpm($context, 'context');
  $settings['extended_valid_elements'] .= ',script[language|type|src]';
}
      verify_html (bool)
      clean up HTML elements that are allowed for valid_elements, extended_valid_elements and invalid_elements
      valid_elements (string)
      there're some but <script> is not allowed
2.4.21.2. Troubleshoot

Fields in node or block configuration might be empty or some elements are missing after save. To see the HTML, disable javascript in Chrome.

2.4.22. IMCE Wysiwyg bridge: imce_wysiwyg

https://www.drupal.org/project/imce_wysiwyg 7.x-1.0 requires imce and wysiwyg

2.4.23. IMCE

requires nothing 7.x-1.10 to 7.x-1.11 IMCE is an image/file uploader and browser that supports personal directories and quota.

2.4.24. ckeditor

requires nothing

2.4.25. nodeblock

create a block for a node

2.4.26. Block Group

blockgroup :: requires core Block d7:blockgroup This module extends the standard drupal block system with block groups. Each block group provides a new block as well as a corresponding region. Child blocks can be moved into any group region. The position and the settings of the parent block are propagated to its children. Also block groups are nestable.

2.4.27. MultiBlock d7:multiblock

multiblock
Built-in in D8. Requires core Block

2.4.28. Conditional Fields

conditional_fields :: d7:module:conditional fields

Conditional Fields for Drupal 7 is an user interface to the new States API, plus the ability to modify fields appearance and behavior on certain conditions when viewing content.

Conditional Fields allows you to manage sets of dependencies between fields. When a field is “dependent”, it will only be available for editing and displayed if the state of the “dependee” field matches the right condition. When editing a node (or any other entity type that supports fields, like users and categories), the dependent fields are dynamically modified with the States API. A simple use case would be defining a custom “Article teaser" field that is shown only if a "Has teaser" checkbox is checked, but much more complex options are available.

It works on AJAX node edit/new form

2.4.29. Search404

search404 :: requires core search

If a user goes to http://example.com/does/not/exist, this module will do a search for "does not exist" and shows the result of the search instead of the 404 page.

2.4.30. Google Analytics

google_analytics :: requires nothing

2.4.31. Elysia Cron

elysia_cron :: Requires nothing. Don't put a cron key in Elysia otherwise the cron run in drush will not work.

2.4.32. SMTP Authentication Support

smtp :: requires nothing

2.4.33. Chaos Tools - ctools

ctools :: no dependencies d7:ctools From 7.x-1.10 to 7.x-1.12

$modulepath/help/ $modulepath/help/plugins-creating.html :: all plugin types

2.4.34. Address Field - addressfield - Package: Fields

Requires d7:ctools

Field List: admin/reports/fields

/moduels/field/field.api.php https://api.drupal.org/api/drupal/modules%21field%21field.api.php/7.x

2.4.35. Email - email - Package: Fields

Requires nothing

2.4.36. Geofield - Package: Fields

Requires d7:geophp d7:ctools Used with d7:geocoder

2.4.37. Viewfield - Package: Fields

Requires d7:views e.g. An order has multiple items, store those items into a field

2.4.38. Field collection - Package: Fields d7:proj:field_collection

  • Any number of fields can be attached to a field-collection field which is an entity
  • d7:entity
  • Test require d7:entity_translation
  • Structure > Field collections

2.4.39. Social Field - socialfield - Package: Fields

Requires nothing

2.4.40. Video Embed Field - video_embed_field - Package: Media

Requires d7:ctools core image New field type Used with video_embed_* modules

Have to install other modules to support different providers e.g. video_embed_facebook, video_embed_brightcove

2.4.41. Display Suite - ds - Package: Display Suite

Requires d7:ctools d7:ds Submoduels

ds_devel
requires devel
ds_extras
admin/structure/ds/list/extras
Field Templates
theme a field. To theme a display field, you don't need to enable it
(no term)
Extra Fields
(no term)
Other
Region to block
Create a region and move fields into that region Content Type > Manage Display > Block regions This will create a block which can be used in other places/regions in Blocks setting
View mode per node
specify a view mode for each node
(no term)
ds_format
(no term)
ds_forms
(no term)
ds_search Integrate search engine such as module apachesolr and d7:search_api_solr to display the search results for a content type
(no term)
ds_ui

https://www.youtube.com/playlist?list=PL7E361A55994F1648

Can display fields of a content type into a layout which has several regions under Manage Display

A display field can be created for a content type display. This field is for theming only not physically create a field in db. Field value can be PHP code and Token can be used.

Field
custom field with PHP and Token
Block field
display a block
Dynamic field
choose a variable to display e.g. node_body, menu
Preprocess field
display a variable e.g. Node Url (machine name node_url)

Later theme that display field using ds_extras

View Mode :: Default, Teaser Create a view mode. d7:entity_view_mode can also create view modes. To display a view that returns multiple nodes of a content type, go to Format > Show > Display suite > Settings Choose the custom view mode and also specify the first item to use View Mode #1 and the second item to use View Mode #2, etc. Custom hook can be selected in this setting to make customization

2.4.42. Meta Tag - metatag

  • requires ctools and token
  • Add metadata tags such as meta description, meta keywords, social media Open Graph

2.4.43. Entity Reference - entityreference

entityreference :: requires d7:ctools d7:entity d7:entityreference In D8 core Add a entity reference field which you can use an entity reference view as a list of values to select from as values in that field. Refer to d7:entity reference view

2.4.44. File Entity

file_entity d7:file entity Requires: d7:ctools, core: Field, Field SQL storage, File

From 7.x-2.0-beta2 to 7.x-2.3

2.4.45. Webform d7:webform

webform
7.x-3.x requires nothing. 7.x-4.x requries d7:ctools d7:views 3

hook_form_alter, hook_form_webform_client_form_alter, (> 7.x-4.4 use hook_form_webform_client_form_NID_alter) Modify in $form['submitted']

2.4.46. Webform Validation

webform_validation :: requires d7:webform

2.4.47. Panels

panels :: drag-and-drop layout for admin Requires: d7:ctools Has a lot of security updates From 7.x-3.6 to 7.x-3.9

2.4.48. Facet API d7:facetapi

  • Requires d7:ctools
  • current_search, facetapi_bonus

2.4.49. Search API - search_api

Requires entity d7:search_api sub modules :: search_api_views (views), search_api_facetapi (d7:facetapi) https://www.drupal.org/docs/7/modules/search-api https://www.youtube.com/watch?v=hcAM0HrEk4c

2.4.49.1. Glossary

For example, a "Node index" for indexing nodes. It would contain the fields that should be indexed and their types, the data alterations and processors to use and some other settings. The details of how to index data are independent of these settings, and server-specific. Index settings are independent of the inner mechanics of the search server.

A server is a concrete way to index and search data. It could, e.g., represent a certain database, a connection to an external search server, etc. How exactly the data is stored is determined by the server's service class, but is not important for the overall functionality of the Search API. A server can have an arbitrary number of indexes attached to it, whose search data is then indexed on that server.

When creating a server, a service class has to be chosen

2.4.49.2. Index

If an index is updated, reindex all content.

  1. Select index fields

    Indexed for those fields for which you want to store data on the search server. These fields can then be searched, used for filtering and sorting, and maybe also used for other purposes.

    Note that only fields of type Fulltext can be used in fulltext searches. So when you want to find individual words contained in this field, not just the whole field value, use this type. Other types can be used, e.g., for filtering and sorting.

    Fields indexed with type "Fulltext" and multi-valued fields (marked with 1) cannot be used for sorting. The boost is used to give additional weight to certain fields, e.g. titles or tags. It only takes effect for fulltext fields.

  2. Add Related Fields

    Items of a certain type might be connected to, or might reference, other kinds of data. For instance, content will always have an author, could also contain references to taxonomy terms, etc. With the Add related fields form at the bottom of the page you can add the fields of those related items to the list, so they can be indexed, too. Use this if you want, e.g., to index the user roles of a node's author. You can also add nested related fields, e.g., the node's author's profile's image's file type.

  3. Customize Workflow (Filters Tab)

    https://www.drupal.org/node/1254452

    1. Data alterations

      Bundle filter Lets you to prevent entities from being indexed based on their bundle (content type for nodes, vocabulary for taxonomy terms, etc.). This way you can, for instance, create an index solely for news. Language control Allows you to control the language of items stored in the index. This is done by providing two different functionalities: Normally, the content of the Item language property (which is automatically added by the Search API for all indexed items) is determined by the item's language property, if available, and otherwise set to undefined. With this data alteration, you can select any other property as an alternative source for the item language, which will then be used instead. Note that the selected field has to contain a single valid ISO language code for each item for this to work, though. You can then also select the languages items in this index may have. Items with any other language (defined by the Item language property) will be rejected during indexing.

      Node access Adds node access checks to searches on this index. This is done by adding a new field, Node access information that stores the relevant access data. When the Node access information, author, and Status fields are present and indexed, appropriate filters will be automatically added to all searches so that they only return results that the current user is allowed to view. Some searches (e.g., search views) provide the option to override this behaviour on a per-search basis, though. Check the corresponding module's documentation for details. In any case, you have to keep in mind that these access checks are solely based on the indexed data. If a node is edited in a way that changes its accessibility (e.g., by being unpublished), this change will only take effect once the node is indexed in its latest state. This means that there is potentially a gap between changing the node and the update of the access checks on search results, meaning that—depending on the data displayed for search results—users could in that time see data that should not be accessible to them. If you need to avoid that, use the index's Index items immediately option.

      Also note that access on the individual fields is never checked — don't include them in the display, if they contain sensitive data. Refer to hook_node_access_records() and hook_node_grants() on implementing node access checks. The node access data stored in the index is based on the node_access table which is affected by hook_node_access_records(). The data alteration is only available for node indexes.

      Search views do not filter based on node access by default. There is a simple option in the query settings called "Additional access checks on result entities" that will do an access check after the actual query is run, but this option should only be used as a last result. Search results counts and facets will not reflect the further restriction applied by views.

      The proper way to do the node access checks in views is to add a filter on Indexed Node: Node access information. This can be complicated because it is important to know what values the field will hold, and this information can not be output through fields in the view itself. One must look to the data stored in the search server. In the case of Solr, this can be accomplished by examining the sm_search_api_access_node field in the schema browser. A sample value for one configuration of taxonomy access control was node_access_taxonomy_access_role:2. One could make a views search display for authenticated users for example that included all results, and a display for anonymous users that checked that the Node Access Information value is not equal to node_access_taxonomy_access_role:2 in the example above. A brief example can be found in this Drupal Answers answer

      URL field Adds a field containing the URL at which the entity can be displayed. For some item types, like nodes, this URL is already available, but this data alteration can be used to also add them for other types.

      Aggregated fields Offers the ability to add additional fields to the entity, containing the data from one or more other fields. Use this, e.g., to have a single field containing all data that should be searchable, or to make the text from a string field, like a taxonomy term, also fulltext-searchable. The type of aggregation can be selected from a set of values: you can, e.g., collect the text data of all contained fields, or add them up, count their values, etc.

      Complete entity view Adds a field containing the whole HTML content of the entity as it is viewed on the site. The view mode used can be selected. This allows you to index exactly „what the user sees“, which is often what is expected, but might differ from just indexing the contents of other fields. Note that this might not work for items of all types. All core entity types except files are supported, though.

      Index hierarchy Allows you to index hierarchical fields along with all their parents. Most importantly, this can be used to index taxonomy term references along with all parent terms. This way, when an item, e.g., has the term New York, it will also be matched when filtering for USA or North America.

    2. Processors

      A processor can do preprocess data that is being indexed, preprocess search queries, and postprocess search results Refer to service class documentation of the server.

      Ignore case Makes searches on selected fields case-insensitive. Some servers might do this automatically, for all others this should probably always be activated, at least for fulltext fields.

      HTML filter Strips HTML tags from selected fields and decodes HTML entities. If you are indexing HTML content (like node bodies) and the search server doesn't handle HTML on its own, this should be activated to avoid indexing HTML tags, as well as to give e.g. terms appearing in a heading a higher boost.

      Tokenizer This processor allows you to specify how indexed fulltext content is split into seperate tokens – which characters are ignored and which treated as white-space that seperates words.

      Stopwords Enables the admin to specify a stopwords file, the words contained in which will be filtered out of the text data indexed. This can be used to exclude too common words from indexing, for servers not supporting this natively.

      Highlighting Adds highlighting of search terms to the search results.

2.4.49.3. Create a search view or a search page

https://www.drupal.org/docs/7/modules/search-api/getting-started/search-forms-and-results-pages/search-api-views Admin > Structure > Views > Add new view (admin/structure/views/add)

View name: Search Show: [the name of the Search index] (in the case of fuzzysearch the default is "Default fuzzysearch index") Create a page [tick] Page title: Search Path: search Display format: Unformatted list of Rendered entity Items to display: 10, use pager Continue & edit (the new View)

Format: Show: Rendered entity | Settings

View mode: Search results Filter criteria

Fulltext search: Expose this filter, Required, Remember the last selection, Use as: search keys Sort criteria

Search: Relevance, descending (if you don’t have an order with fuzzysearch you will get a PDO exception) Page settings

Access: Permission: view published content Advanced

No results behaviour: Global: Text area “No results matched your search.” Exposed form

Exposed form in block: Yes (This option will only show up on Page displays.) Exposed form style settings: Submit button text: Search Save the View. Add the exposed form block to a region. Note that you will only receive results for partial matches that are longer than the minimum word length specified in the Index configuration.

Views based on Search API use as base table a search_api_index_* table, not the usual node table or other ones supported by Views. That is why the Search views are selected at the views' creation moment, and it can not be changed later.

2.4.49.4. Convert an existing view to search api

If the views to be converted use the row plugins Content (node), Fields or Rendered entity, the conversion may be easier (see further). If not (the selected row plugins were others), then unfortunately the views should be fully recreated manually: create a brand new search_api view and recreate all the elements from your old view by hand.

For views using Content, Fields or Rendered entity row plugins: Export the view to be converted. Save this code as a backup. Modify the view, removing all the Filters (contextual or not), all the Sort criteria and all the Relationships. Write down the removed items: you will need to recreate them manually using the indexed versions. When the view is clean of filters and sort criteria, export it again. In the export code, the 5th line will contain the $view->base_table variable. Change it to the search api index to be used in the view (the index must have been created before):

$view->base_table = 'search_api_index_<name_of_your_index>'; Maybe fiddle around with fields (see below). Go to admin/structure/views/import and import the modified code. Do not forget to select the option Replace an existing view if one exists with the same name. (NOTE: currently there is a bug in Ctools 1.x that makes the Import option not to be shown for administrator users different than uid=1, here is the issue: #870938: Add new permission for controlling imports ). Add the previously removed Filters, Sort criteria and Relationships, selecting them from the now available indexed versions. Process will be easy if the views to be converted used the Rendered entity row plugin, since no changes will be required for the row contents.

Field fiddling

For fields there is no general recipe, although these hints should catch most cases:

Care that your index contains all needed fields or create them when import errors bug you. Look for lines that look like this: $handler->display->display_options['fields']['field_FOO']['table'] = 'TABLE' If TABLE is 'views', don't change. If TABLE is 'node' (or your base table) or 'field_data_FOO', change to 'search_api_index_<name_of_your_index>'

2.4.49.5. Configure Facet, Facet Blocks

A block is created for each facet configured. Even though it's set to appear on all pages, it won't display for a search view or a search page.

2.4.49.6. Views Filter > Search: Fulltext search
2.4.49.7. Current Search Block

current_search is installed with d7:facetapi. /admin/config/search/current_search

2.4.49.9. Developer Documentation

https://www.drupal.org/docs/7/modules/search-api/developer-documentation https://www.drupal.org/docs/7/modules/search-api/advanced-site-building-tutorials

  1. Execute a search in code
    // Replace "node_index" with you index's machine name.
$query = search_api_query('node_index');

$query->condition('type', 'product', '=');
$filter = $query->createFilter('OR');
$filter->condition('field_category', 'book', '=');
$filter->condition('field_category', 'magazine', '=');
$query->filter($filter); 

$data=$query->execute();
$results=$data['results'];
print_r($results);
  2. Combined fields to do IN or OR across multiple fields

    https://www.drupal.org/docs/7/modules/search-api/advanced-site-building-tutorials/combining-fields https://www.drupal.org/project/search_api_combined

    Limitation At present the module has been tested for multiple term references and not with any other field type. It is currently set to index all combined fields as lists of items so it will not work with multiple fields that have single values that expect to stay as single values – where this is important is in sorting, for instance, since you cannot sort by multiple values with Solr.

2.4.49.10. API of Search API

http://www.drupalcontrib.org/api/drupal/contributions%21search_api%21search_api.api.php/7 path-to-mod/search_api/search_api.api.php

Search search_api*.api.php search_api_solr.api.php

2.4.50. Search API Solr - search_api_solr - Package: Search

search_api_solr requires d7:search_api and core Search d7:search_api_solr It's more powerful than module apachesolr It creates View Mode called Search Index in each content type

Setup an account and create an index on Opensolr.com

On OpenSolr UI, upload the 4.x or corresponding Solr version based on OpenSolr version. path-to-mod/search_api_solr/solr-conf/4.x https://opensolr.com/blog/2011/09/how-to-use-with-drupal

Lucene version :: search lucene in solr-conf/4.x, e.g. solr.luceneMatchVersion=LUCENE_40 <luceneMatchVersion>${solr.luceneMatchVersion:LUCENE_40}</luceneMatchVersion>

In bundle (e.g. node type abc), setup Manage Display for the Search Index.

2.4.50.1. Debug

In OpenSolr UI for an index (tnt_dev), there's a button Browse Data. https://path-to-aws.opensolr.com/solr/tnt_dev/select?q=*:*&wt=json&indent=true&start=0&rows=5

Insert a key-value pair to search https://path-to-aws.opensolr.com/solr/tnt_dev/select?q=im_field_industry:337&wt=json&indent=true&start=0&rows=5

Retrieve the request made to Solr and response from Solr In search_api_solr/includes/solr_connection.inc add a line below protected function makeHttpRequest 

drupal_set_message(check_plain($url)); // <-- Add this line to retrieve request
$result = drupal_http_request($url, $options);
drupal_set_message(check_plain($result->data)); // <-- Add this line to retrieve response

2.4.51. Search API Solr Overrides - search_api_solr_overrides

Requires d7:search_api_solr Provides site/environment specific overrides for search_api_solr configuration in settings.php

2.4.52. Views

Views, sub module Views UI d7:views d7:viewsapi Requires d7:ctools From 7.x-3.12 to 7.x-3.16

2.4.53. Better Exposed Filters

better_exposed_filters d7:better exposed filters Requires d7:ctools d7:views Group under Views 7.x-3.3, 7.x-3.4

2.4.54. Views Bulk Operations

views_bulk_operations : sub module actions_permissions d7:views_bulk_operations Requires: d7:entity d7:views

2.4.55. Administration views

admin_views: d7:views d7:views_bulk_operations

2.4.56. Media

media Requires: d7:file entity, d7:ctools, d7:views

From 7.x-2.0-beta1 to 7.x-2.9

2.4.57. Internationalization - i18n

i18n requires core locale and d7:m:variable

d7:m:i18n

Add a language
admin/config/regional/language
(no term)
Go to pages that use t('string to translate', [], ['langcode'=>"fr"]) with non default options

Once a language is added (default language is chosen), node created will have $node->language = 'en'

2.4.58. S3 File System - s3fs d7:m:s3fs

  • https://pantheon.io/docs/drupal-s3/
  • Requires module libraries, library AWS SDK for PHP 2.x, PHP 5.3.3+ with allow_url_fopen = On in php.ini
  • It replaces public:// or private:// with s3://
  • After this is enabled, files that are in Drupal but not in S3 will become unavailable! Drupal files need to be uploaded to S3 first!
  • Without replacing public:// with s3://, only new files will be uploaded to S3. If a node has files in public:// and they are not uploaded to s3:// and without replacing public:// is set, saving the node will not change the files from public:// to s3://. In short, it's safe to not enable replace public:// with s3://
  • drush s3fs-copy-local. Refer to module README.txt for more info
  • In order to sync files from S3 to Drupal, Refresh file metadata cache is needed /admin/config/media/s3fs/actions. Best to run drush in case of timeout drush s3fs-refresh-cache
drush dl s3fs
drush en s3fs
# update.php is not necessary

# Download library
# In Pantheon,
drush make --no-core code/sites/all/modules/s3fs/s3fs.make code

# Or 
drush make --no-core sites/all/modules/s3fs/s3fs.make
  • Check if library is downloaded sites/all/libraries/awssdk2/aws-autoloader.php
  • /admin/config/media/s3fs/settings or in settings.php
  • Set Cache-Control header to be public, max-age=604800
  • Refer to header:cache-control

settings.php 

$conf['awssdk2_access_key'] = 'YOUR ACCESS KEY'; 
$conf['awssdk2_secret_key'] = 'YOUR SECRET KEY'; 
$conf['s3fs_root_folder'] = 'thenyourroot'; // bucketroot/thenyourroot/*
$conf['s3fs_bucket'] = 'YOUR BUCKET NAME'; 
$conf['s3fs_region'] = 'YOUR REGION'';
Set Default download method to Amazon Simple Storage Service
Go to admin/config/media/file-system
Set Upload destination for a file field for a content type
Strucutre > Content Types > choose one > choose the file field
(no term)
Refer to aws:s3 about how to setup user with correct permission policy
2.4.58.1. IMCE & S3FS, TinyMCE

If S3fs is < 7.x-2.5 then get the latest s3fs dev version.

Any field of type File, Image, etc. can set the "Upload destination" to S3 in the Field Settings.

To insert javascript inside Wysiwyg with TinyMCE (v3.5.8), first in admin allow user to turn off TinyMCE by default.

/admin/config/content/wysiwyg/profile/full_html/edit, Basic Setup, Allow users to choose default

Login to the user profile, and turn off Text formats enabled for rich-text editing > Full HTML

That user can still enable rich-text but the default is plain text without stripping script tags.

However, toggle rich-text on and off still strips off script tags.

In short, it's hacky to enable users to insert javascript. You may need the shortcode module.

2.4.59. Password Policy - password_policy

Change password required length, etc.

2.4.60. Secure Login - securelogin

It enforces secure authenticated session cookies and ensures the user login page, any pages with the user login block and other forms you configure are submitted securely via HTTPS.

2.4.61. Secure Review - security_review

Automate testing

2.4.62. Security Kit - seckit

  • https://www.drupal.org/project/seckit
  • Refer to header:csp
  • Content Security Policy (CSP) implementation via HTTP response header Content-Security-Policy
  • Control over Internet Explorer / Apple Safari / Google Chrome internal XSS filter via X-XSS-Protection HTTP response header
  • Prevent content upsniffing and serving files with incorrect MIME-type via X-Content-Type-Options: nosniff HTTP response header

2.5. Debug

Enable module Database Logging and Syslog if necessary.

watchdog('LOG_TYPE','message'); watchdog('LOG_TYPE', $message, array(), WATCHDOG_ERROR); array() is to t() $message. Don't forget to put empty array otherwise you will have to manually clean up the watchdog table!

If error or warning messages are printed using drupal_set_message, modify the `drupal_set_message` function so that it prints debug_backtrace 

if ($type == 'error') {
  $message .= ' '. print_r(debug_backtrace(DEBUG_BACKTRACE_PROVIDE_OBJECT,3),1);
}

You can throw an Exception 

throw new Exception(print_r($val));

if (ip_address() === '123.456.789.123') {
  var_dump();
}

// can be used to see what files include this file
var_export(debug_backtrace());

// Just return the last called function
$dbt = debug_backtrace(DEBUG_BACKTRACE_IGNORE_ARGS,2);
$caller = isset($dbt[1]['function']) ? $dbt[1]['function'] : null;

2.5.1. debug_backtrace

  • $options
    DEBUG_BACKTRACE_PROVIDE_OBJECT
    default. Include everything
    DEBUG_BACKTRACE_IGNORE_ARGS
    Don't include function/method arguments
  • index 0 being the most recently run code, the last index is the entrypoint
    • Can be used to see which parent file includes a file
    • If the current calling place is the first script file that is run, it will return an empty array
debug_backtrace(int $options = DEBUG_BACKTRACE_PROVIDE_OBJECT, int $limit = 0): array

2.6. Configuration, Setting, Maintenance Mode

Override configuration and setting based on the current dev environment settings.php 

$conf['admin_theme'] = 'seven';
$conf['maintenance_mode'] = FALSE;
To determine which environment
pantheon:environment

When site is under maintenance and you want to login:

  • Turn off javascript in browser
  • a.com/?q=user
  • UPDATE `variable` SET `value`='0' WHERE `name` = 'maintenance_mode'

For assigning permissions to roles, e.g. enable anonymous for Devel, you have to do it manually

Variable name and its default value

admin_theme
'seven'
preprocess_css
"1"/"0", aggregate.
preprocess_js
"1"/"0", aggregate.
cache
"1"/"0", cache pages for anonymous users
block_cache
"1"/"0", cache blocks
cache_lifetime
"0", none. Minimum cache lifetime
error_level
"0""1""2", none/error and warnings/all messages. Logging and errors.

If setting is an array, you have to override the whole array. Can't override a key value.

2.6.1. d7:mysql:charset

  • Since Drupal 7.50, Drupal now supports 4 byte UTF-8 with MySQL. You will first need to run a custom drush command provided by a utf8mb4_convert module to convert all Drupal database, tables and fields to charset utf8mb4 and collation utf8mb4_general_ci. Then change the database connection settings in settings.php
  • Drupal Doc
  • The utf8mb4_convert module requires inno_large_prefix=true when MySQL server is bootup, Otherwise a test ->utf8mb4IsSupported can't pass before any actions

    • To bypass the test, re-create any string index columns varchar(255) to varchar(191) 

      ALTER TABLE cache_block CHANGE cid cid VARCHAR(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
    • Do this for any problematic column the module returns
  • For new website, changing the database connection settings in settings.php is good enough
  • If MySQL server doesn't bootup with inno_large_prefix=true, all indexed columns will have max varchar(191)
  • As a quick workaround, you can remove all php:4byte characters
  • Refer to mysql:charset for important info

2.6.2. Show error d7:show error

See 2.5 

error_reporting(E_ALL);
ini_set('display_errors', TRUE);
ini_set('display_startup_errors', TRUE);
$conf['error_level'] = 2;
// UI > Administration > Configuration > Development > logging > 'All messages'

2.6.3. Session Timeout, Session Cookie Timeout

// these 2 lines are to make sure PHP has the settings to enable gc
ini_set('session.gc_probability', 1);
ini_set('session.gc_divisor', 100);

// about 2 days for session timeout
ini_set('session.gc_maxlifetime', 200000);

// about 20 days for session cookie timeout
ini_set('session.cookie_lifetime', 2000000);

2.7. URL Alias

Setup pattern for each node type
Configuration > Search and metadata > URL Aliases > Patterns

2.8. Link, Module Path, Theme Path, File Path

2.8.1. Link

Empty link! 

// Basic syntax: l($text, $path, array $options = array())
l($text, 
  '', 
  array('fragment'=>' ', 
    'html'=>TRUE, // if $text is HTML, set this to true
    'external'=>TRUE,
  //    'attributes' => array(
  //       'target'=> '_blank'
  //       'class' => ['classA'],
  //     ),
  )
);
  • Function l add class="active" when the path is current_path(). Use url() instead.
url($path = NULL, $options = array());
return
string of URL
$path
internal or external e.g. node/34 http://example.com/foo http://a.com/?foo=bar
$options
append components
query
an array of query key/value-pairs (without any URL-encoding) to append to the URL. Result is URL encoded
fragment
A fragment identifier (named anchor) to append to the URL. Do not include the leading '#' character.
url($path = NULL, $options = array())

Get URL alias url() just returns a url. $options are the same as l(). url('taxonomy/term/1');

Embed a url directly to HTML href
check_url($url)
Embed a url directly as a URL parameter
echo 'http://abc.com/?t='.urlencode($url);

2.8.2. Current External URL, d7:url d7:current_path

url(current_path(), array('absolute' => TRUE));

;; With query parameters
url(current_path(), array('absolute' => TRUE, 'query' => drupal_get_query_parameters()));

current_path still works when 404. Don't var_dump or print current_path() directly as it might have XSS

url

absolute
append http
base_url
override the default $base_url with a custom base url, e.g. http://abc.com without trailing stash

2.8.3. Module Path, Theme Path, File Path

global $base_url;
$mod_path =  $base_url.base_path().drupal_get_path('module', 'lili_mod_name');
$theme_pat = $base_url.base_path().drupal_get_path('theme', 'lili_theme_name');

// web root path
echo filemtime(getcwd().'\sites\all\themes\express\css\styles.css');

Both have no trailing slash

2.8.4. global $base_url

No trailing slash http://abc.com

2.9. Translate

2.9.1. t()

$text = t("This is !name's website", array('!name' > $username)); $text = t("This is @name's website", array('@name' => $username)); $text = t("This is %name's website", array('%name' => $username)); $text = t(check_plain($text)); $link = l(t("Link text"), "node/123"); $link = t('Visit the <a href"@url">settings</a> page', array('@url' => url('admin')));

/ `%` and `@` use `check_plain()` / `%` adds `<em class="placehoder"></em>` // `!` straight output.

t('Thursday',[],['langcode'=>'fr']); t('Thursday',[],['context'=> 'some context', 'langcode'=>'fr']);

d7:m:i18n

2.9.2. Translate Date and Time

// Have to load all terms then Translate Interface can show up admin/config/regional/translate/translate

$_fr_days_translation = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday'];
$_fr_months_translation = ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December'];
foreach ($_fr_days_translation as $_t_i) {
  t($_t_i, [],['langcode'=>'fr']);
}

// format_date with F (long month name) adds context..
foreach ($_fr_months_translation as $_t_i) {
  t($_t_i, [],['context'=>'Long month name','langcode'=>'fr']);
}

$_u_time = mktime (11, 0, 0, $_u_time['month'], $_u_time['day'], $_u_time['year']);
$_fr_time = format_date($_u_time, 'custom', 'l j F Y', NULL, 'fr');

echo $_fr_time;

2.10. Global Variables

DRUPAL_ROOT
/index.php
VERSION
7.67
(no term)
https://api.drupal.org/api/drupal/globals/7.x
$base_url
'http://www.yoursite.com' wihtout trailing slash
$base_path
$base_root
$databases
$cookie_domain
$conf
$installed_profile
$upodate_free_access
$db_url
$db_prefix
$drupal_hash_salt
$is_https
$base_secure_url
$base_insecure_url
(no term)
Drupal Properties
  • $_SERVER['*']
    QUERY_STRING
    query string without q e.g. 'foo=bar&koo=hoo'

2.11. Common Functions d7:functions

2.11.1. drupal_add_html_head

drupal_add_html_head($tag, 'unique-name');

// $head in html.tpl.php
$tag = array(
  '#tag'        => 'meta',
  '#attributes' => array(
    'property' => 'og:url',
    'content'  => $base_url,
  ),
);
drupal_add_html_head( $meta_og, 'lili_og_url' );

2.11.2. drupal_set_title

drupal_set_title($title = NULL, $output = CHECK_PLAIN);

$title
if NULL, leave the current unchanged

2.11.3. drupal_goto

drupal_goto($path = '', array $options= array(), $http_response_code= 302)

$path
a drupal path 'node/123' or a full url
$options
URL optoins to pass to url()

Redirect to homepage drupal_goto('<front>');

2.11.4. drupal_valid_path($path, $dynamic_allow= TRUE)

$_path = current_path();

if (drupal_valid_path($_path)) { // Current user has access to this path/page }

2.11.5. drupal_clean_css_identifier

Convert string to CSS ready element name, class and ID in selectors. 

echo drupal_clean_css_identifier(drupal_strtolower($class));

2.11.6. Sanitization functions d7:functions:sanitization

https://api.drupal.org/api/drupal/includes%21common.inc/group/sanitization/7.x

check_plain($text)
refer to PHP htmlspecialchars. Can be used to sanitize HTML attribute value. d7:functions:check_plain
drupal_strip_dangerous_protocols($uri)
not for display.
drupal_attributes(array $attributes = [])
Each array key and its value will be formatted into an attribute string. Attr. values are check_plain() but not the key (attribute name).
check_url($uri)
strip and encode a uri for output to HTML. eq. to check_plain(drupal_strip_dangerous_protocols)
filter_xss($string, $allowed_tags = ['a', 'em', …])
Filtes HTML to prevent XSS vulnerabilities.

2.12. Theme API

2.12.1. render() vs drupal_render()

render(&$element) render is a wrapper for durpal_render which

  • makes sure the element passed in is set to be shown show()
  • only renders array. If what's passed in is not an array, it will return as it is.

show() :: $element['#printed'] = FALSE;

drupal_render(&$elements)

  • if #access == false, return ''
  • if #printed = true, return ''
  • if #cached is set, load and return cache drupal_render_cache_get($elements)
  • #markup is set but not #type, #type is set to markup
  • array of pre_render function names to modify
  • pre_render functions might set #printed to some value to indicate not to print, if so return ''
  • get children $children and initialize #children
  • if #theme is set, theme all children and save to #children
  • if #children is still empty, concatenate drupal_render for each child
  • if #theme_wrappers is defined, run each theme_wrapper function for #children
  • if #post_render is defined, run each function for #children
  • if #states is defined, attach JavaScript state drupal_process_states
  • if #attached is defined, drupal_process_attached
  • #prefix . #children . #suffix
  • drupal_render_cache_set
  • #printed set to true

2.12.2. theme()

2.12.3. System Theme

  • Default theme files *.tpl.php are located at /modules/mod_name/mod-name-component-name.tpl.php
  • e.g. /modules/node/node.tpl.php
  • For custom theme, better to put *.tpl.php files under /sites/all/themes/theme_name/templates/

d7:View_Theming

2.12.3.1. Fast detect if it's a node page

if (arg(0)=='node') { $node_id = arg(1); }

2.12.3.2. Include file in *.tpl.php

include(drupal_get_path('theme', 'theme_name').'/example-file-template.tpl.php');

If the file is in the same folder <?php include 'abc.php'; ?>

2.12.3.3. .info (required)
2.12.3.4. html.tpl.php

Get node if it's a node page $node = menu_get_object(); if (isset($node->type)) { … }

2.12.3.5. page.tpl.php

Get node if it's a node page hook_preprocess_page(&$variables) { if (!empty($variables['node']) && $variables['node']->type == 'NODETYPE') { $variables['greeting'] = 'Custom Greeting'; } }

Available variables in page.tpl.php $node

2.12.3.6. region.tpl.php
2.12.3.7. block.tpl.php
2.12.3.8. node.tpl.php d7:template:node

$node $type :: node type

Node status variables $view_mode :: 'full'

  1. Check if a field is empty

    d7:api:field:hook_field_is_empty

  2. $content['field_name']

    print render($content['field_name']) is based on the view mode that the entity is currently used to display and then the field formatter.

    If field_name is not a field of the $node, then $content['field_name] is not set. If field_name is defined but it has no value, then $content['field_name'] is not set. So you can't directly render($content['field_name']) because the field_name might not be defined or has value. 

    render($content['field_name']);

//Raw value
$delta = 0; // first value. Even for text field
$item = $content['field_name']['#items'][$delta]
$item['target-id'] // for entity reference field
$item['entity']->title // entity title
$aField = $item['entity']->field_entity_field_name // array
$aField['und'][0]['value'] // raw value of a field of that entity
$aField['und'][0]['safe_value']
2.12.3.9. field.tpl.php
2.12.3.10. comment-wrapper.tpl.php
2.12.3.11. comment.tpl.php
2.12.3.12. template.php
2.12.3.13. logo.png
2.12.3.14. screenshot.png

2.12.4. Form Theme

$form['#theme'][] = 'lili_form'; If no custom theme function, d7:hook_theme d7:theme_*, is defined, then the default template file is /sites/all/themes/your_theme/templates/form/lili_form.tpl.php

2.12.5. Region

Region can hold multiple blocks. If module d7:blockgroup is installed, a block group is a region but it can be included in other region or block group. Define a region in mytheme.info e.g. regions[header] = Header Render a region in template files print render($page['header'])

2.12.6. Block

2.12.6.1. hook_block_info
function hook_block_info() {
  $blocks             = array();
  $blocks['my_block'] = array(
    'info'  => t( 'My Custom Block' ),
    'cache' => DRUPAL_CACHE_PER_ROLE,
  );

  return $blocks;
}

Use `DRUPAL_NO_CACHE` for development, then `DRUPAL_CACHE_GLOBAL`. Default is `DRUPAL_CACHE_PER_ROLE`

If `Cache Block` is disabled and unchecked in Drupal Performance setting, all blocks will not be cached. Refer to d7:cache block

The delta `my_block` only needs to be unique within the module. HTML ID is `block-MODULENAME-DELTA`

Now the block can be seen in the default theme Blocks admin page /admin/structure/block Place the new block to a region. To assign a block to multiple regions, install d7:MultiBlock module: Then go /admin/structure/block/instances to create another instance of the newly created block type.

2.12.6.2. hook_block_view
<?php
/**
 * Implements hook_block_view().
 */
function lili_block_view( $delta = '' ) {
  $block = array();

  switch ( $delta ) {
    case 'my_block' :
      $block['content'] = _lili_my_block_view();
      break;
  }

  return $block;
}

/**
 * Custom function to assemble renderable array for block content.
 * Returns a renderable array with the block content.
 * @return
 *   returns a renderable array of block content.
 */
function _lili_my_block_view() {
  $block = array();

  // Capture the image file path and form into HTML with attributes
  $image_file = file_load( variable_get( 'block_image_fid', '' ) );
  $image_path = '';

  if ( isset( $image_file->uri ) ) {
    $image_path = $image_file->uri;
  }

  $image = theme_image( array(
    'path'       => ( $image_path ),
    'alt'        => t( 'Image description here.' ),
    'title'      => t( 'This is our block image.' ),
    'attributes' => array( 'class' => 'class_name' ),
  ) );

  // Capture WYSIWYG text from the variable
  $text = variable_get( 'text_variable', '' );

  // Block output in HTML with div wrapper
  $block = array(
    'image'   => array(
      '#prefix' => '',
      '#type'   => 'markup',
      '#markup' => $image,
    ),
    'message' => array(
      '#type'   => 'markup',
      '#markup' => $text,
      '#suffix' => '',
    ),
  );

  // directly return a form as block
  // return drupal_get_form('lili_another_form');
  // or return ['some thing' => drupal_get_form('lili_another_form')];
  return $block;
}
2.12.6.3. hook_block_view_alter
function lili_block_view_alter( &$data, $block ) {
  if ( $block->module == 'block' && $block->delta == '3' ) {
    // $data['subject']: title of the block

    $data['content'] = '';
  }
}
2.12.6.4. Configurable Block: hook_block_configure, hook_block_save

hook_block_configure 

/**
 * Implements hook_block_configure().
 */
function lili_block_configure( $delta = '' ) {
  $form = array();

  switch ( $delta ) {
    case 'my_block' :
      // Text field form element
      $form['text_body'] = array(
        '#type'          => 'text_format',
        '#title'         => t( 'Enter your text here in WYSIWYG format' ),
        '#default_value' => variable_get( 'text_variable', '' ),
      );

      // File selection form element
      $form['file'] = array(
        '#name'              => 'block_image',
        '#type'              => 'managed_file',
        '#title'             => t( 'Choose an Image File' ),
        '#description'       => t( 'Select an Image for the custom block.  Only *.gif, *.png, *.jpg, and *.jpeg images allowed.' ),
        '#default_value'     => variable_get( 'block_image_fid', '' ),
        '#upload_location'   => 'public://block_image/',
        '#upload_validators' => array(
          'file_validate_extensions' => array( 'gif png jpg jpeg' ),
        ),
      );
      break;
  }

  return $form;
}

/**
 * Implements hook_block_save().
 */
function lili_block_save( $delta = '', $edit = array() ) {
  switch ( $delta ) {
    case 'my_block' :
      // Saving the WYSIWYG text
      variable_set( 'text_variable', $edit['text_body']['value'] );

      // Saving the file, setting it to a permanent state, setting a FID variable
      $file         = file_load( $edit['file'] );
      $file->status = FILE_STATUS_PERMANENT;
      file_save( $file );
      $block = block_load( 'lili', $delta );
      file_usage_add( $file, 'lili', 'block', $block->bid );
      variable_set( 'block_image_fid', $file->fid );
      break;
  }
}
2.12.6.5. Block API Sequence

hook_block_configure: Define a configuration form for a block. hook_block_info: Define all blocks provided by the module. hook_block_info_alter: Change block definition before saving to the database. hook_block_save: Save the configuration options from hook_block_configure(). block_list

  • _block_load_blocks
    • hook_block_list_alter
  • _block_render_blocks: render a set of blocks for one region. Check if it's cacheable, then get and set cache_block
    • _block_get_cache_id
    • hook_block_view: Return a rendered or renderable view of a block.
    • hook_block_view_alter: Perform alterations to the content of a block.
    • hook_block_view_MODULE_DELTA_alter: Perform alterations to a specific block.
2.12.6.6. Render a block

If a block is assigned to a region "content" Markup of the the block is $page['content']['block_123'] 

$SIDEBAR_CONTENT = json_encode($page['content']['block_'.$block_id]);
$SIDEBAR_CONTENT = json_decode($SIDEBAR_CONTENT,true);
$SIDEBAR_CONTENT = $SIDEBAR_CONTENT['#markup'];

2.12.7. Custom Theme Function

Module theme: theme('modname_theme-name',$data) d7:hook_theme

$existing
array. existing themes
$type
string. Whether a theme, module, etc. is being processed. e.g. Is it a parent theme?
$theme
the name of theme, module, etc. is being processed
$path
the directory path of theme or module

Return an array

variables
theme function takes no parameter array(). Assign null to key-value for default values.
path
Use with 'template'
  • If theme_* function is used, you don't need to specify 'path' and 'template'
  • Whether or not hook_theme is defined in template.php or in a module, the path default is to the module or theme
  • But you should make the path to sites/all/themes/your_theme/templates/ or sites/all/modules/mod_path/modname/templates
  • e.g. 'path' => drupal_get_path('module', 'panels') . '/templates', 'path' => drupal_get_path('theme', 'panels') . '/templates',
  • if path is not defined but template is defined as a path to file (theme/template_file_name), then
    hook_theme defined in template.php
    path_to_theme/theme/template_file_name.tpl.php
    hook_theme defined in a module
    path_to_module/theme/template_file_name.tpl.php
  • if path is not defined but template is defined as a file (template_file_name), then
    hook_theme defined in template.php
    path_to_theme/template_file_name.tpl.php
    hook_theme defined in a module
    path_to_module/template_file_name.tpl.php
template
  • Assuming the theme function name is theme_lili_flair
  • Default file name is lili_flair and the actual file is lili_flair.tpl.php

d7:theme_* 

function lili_theme( $existing, $type, $theme, $path ) {
  return array(
    'lili_flair' => array(
  'variables' => array(
    'text' => NULL,
  ),
    ),
  );
}

function theme_lili_flair( $variables ) {
  if ( ! empty( $variables['text'] ) ) {
    return '<span class="li-tag li-tag-pill li-tag-color">' . $variables['text'] . '</span>';
  } else {
    return '';
  }
}

echo theme('lili_flair',array('text'=>'hi'));

// In xxx.tpl.php use $text directly

2.12.8. Sub theme

  • https://www.drupal.org/docs/7/theming/creating-a-sub-theme
  • The name of your sub-theme must start with an alphabetic character and can only contain lowercase letters, numbers and underscores
    Folder
    my_subtheme
    .info file
    name = My Subtheme
  • base theme = theme_name
  • The sub-theme inherits most properties of the base theme. The important exceptions are
    • regions
    • core version
    • color info
    • features
    • parent theme's logo (logo.png/logo.jpg)
    • parent theme's favicon.ico
    • theme-settings.php
  • May want to copy the regions section of your base theme's info file, along with its core version declaration

  • If your base theme supports the color module and you'd like your sub-theme to support it, you probably also want to copy the color folder from your base theme and add the line from your base theme's info file to your sub-themes info file that looks like: 

    stylesheets[all][] = css/colors.css

    and then copy the colors.css from base theme to the css folder of the sub-theme.

  • Style sheets and JavaScripts are inherited. If you want to override, create relevant files and add these to .info 

    stylesheets[all][] = style.css
scripts[] = script.js
features[] = logo
  • template.php funciton inheritance
    • Anything defined in the parent theme's template.php file will be inherited. This includes theme function overrides, preprocess functions and anything else in that file
    • Each sub-theme should also have its own template.php file, where you can add additional functions or override functions from the parent theme
  • theme functions, e.g. theme('[hook]', $var,...), When a sub-theme overrides a theme function, no other version of that is called
  • preprocess functions, e.g. [theme]_preprocess_page is called before page.tpl.php is rendered
    • Unlike theme functions, preprocess functions are not overriden in a sub-theme
    • Instead, the parent theme preprocess function will be called first, and the sub-theme preprocess function will be called next
    • There is no way to prevent all functions in the parent theme from being inherited. The only way to remove a parent themes' preprocess function is through hook_theme_registry_alter()
  • Page, node, block and other template (.tpl.php) file inheritance
    • node--blog.tpl.php building on an inherited node.tpl.php
    • A single hyphen is still used to separate words. The double hyphen always indicates a more targeted override of what comes before the --. e.g. node--long-content-type-name.tpl.php
  • Add a template file with the same name in sub-theme folder to have it override the template from the parent theme
  • Screen shots inheritance
name = Jean
description = A subtheme of Bartik, which is a flexible, recolorable theme with many regions.
core = 7.x
base theme = bartik

; Add a style sheet for all media
stylesheets[all][] = css/jean.css
stylesheets[all][] = css/colors.css

; Add a style sheet for screen and projection media
; stylesheets[screen, projection][] = theScreenProjectionStyle.css
; Add a style sheet for print media
; stylesheets[print][] = thePrintStyle.css
; Add a style sheet with media query
; stylesheets[screen and (max-width: 600px)][] = theStyle600.css
; avoid using *_style.css but *style.css e.g. myownstyle.css is ok..

regions[header] = Header
regions[help] = Help
regions[page_top] = Page top
regions[page_bottom] = Page bottom
regions[highlighted] = Highlighted

regions[featured] = Featured
regions[content] = Content
regions[sidebar_first] = Sidebar first
regions[sidebar_second] = Sidebar second

regions[triptych_first] = Triptych first
regions[triptych_middle] = Triptych middle
regions[triptych_last] = Triptych last

regions[footer_firstcolumn] = Footer first column
regions[footer_secondcolumn] = Footer second column
regions[footer_thirdcolumn] = Footer third column
regions[footer_fourthcolumn] = Footer fourth column
regions[footer] = Footer

settings[shortcut_module_link] = 0
  • General steps to create a sub theme
    Create the folder
    /sites/all/themes/jean
    Create .info file
    /sites/all/themes/jean/jean.info
    Create a blank file named
    /sites/all/themes/jean/css/jean.css
    Copy from bartik or create your own
    /sites/all/themes/jean/logo.png
    (no term)
    In order to get the color module to work with your subtheme, you will need to do the following:
    • Copy the file /themes/bartik/css/colors.css to /sites/all/themes/jean/css/colors.css
    • Copy the folder and its contents /themes/bartik/color/ to /sites/all/themes/jean/color/
    (no term)
    Go to the Administration > Appearance page to enable your new subtheme called Jean. Now you can add CSS to your jean.css file, and it will apply to your new subtheme

2.12.9. 404 page

page–404.tpl.php could be in theme folder or theme's templates folder 

function jean_preprocess_page(&$vars) {
  $header = drupal_get_http_header('status');
  if ($header == '404 Not Found') {
    $vars['theme_hook_suggestions'][] = 'page__404';
  }
}

2.12.10. Contributed themes

2.12.11. Render a field of a node, a term

2.12.11.1. field_get_items, field_view_value, field_view_field

field_view_field returns a renderable array that has everything: lable and value. field_view_value returns a renderable array that only has the formatted value. Such as text field simply returns text 

$delta = 0; // first value
$field = field_get_items('node', $node, 'field_youtube_link');
// 
$output = field_view_value('node', $node, 'field_youtube_link', $field[$delta]);
// $output is an array
print render($output);

// it seams field_view_value can't render node title

$output 

[ 
  "#markup" => "...",
  "#access" => true
]

Taxonomy term field 

$eventField = field_get_items('node', $node, 'field_event_category');
$eventTerm = field_view_value('node', $node, 'field_event_category', $eventField[0]);
$titleImLookingFor = $eventTerm['#title'];

Change value before rendering 

$image = field_get_items('node', $node, 'field_image');
$output = field_view_value('node', $node, 'field_image', $image[0], array(
  'type' => 'image',
  'settings' => array(
    'image_style' => 'thumbnail',
    'image_link' => 'content',
  ),
));
2.12.11.2. Using node template

In page.tpl.php 

$node_view = node_view($node);
// different view mode, default is 'full'
// node_view($node, 'custom-view-mode');
// different language, default is the global content language of the current request
// node_view($node, 'full', 'fr');
print drupal_render($test);

Add a view mode 

/**
 * Implements hook_entity_info_alter().
 */
function MODULE_entity_info_alter(&$entity_info) {
  $entity_info['node']['view modes']['block_feature'] = array(
    'label' => t('Block feature'),
    'custom settings' => TRUE,
  );
}

/**
 * Implements hook_preprocess_node().
 */
function MODULE_preprocess_node(&$variables) {
  if($variables['view_mode'] == 'block_feature') {
    $variables['theme_hook_suggestions'][] = 'node__' . $variables['type'] . '__block_feature';
  }

  // add css files
  $node = $variables['node'];
  if (in_array($node->type, ['nodetype1','nodetype2','nodetype3'])) {
    drupal_add_css(drupal_get_path('theme', 'mytheme') . "/css/owl-carousel/owl.carousel.min.css");
    drupal_add_css(drupal_get_path('theme', 'mytheme') . "/css/owl-carousel/owl.theme.default.min.css");
  }
}

Get raw value 

echo $content['field_youtube_link']['#items']['0']['value'];

2.13. Drush

2.13.1. version drush version

2.13.2. Debug -vd

2.13.3. Alias

drush sa list all aliases pantheon:drupal:alias

2.13.4. User

# one time login link. uid, user name, or email address for the user. Default is uid 1
drush user-login yourusername
drush user-create newuser --mail="a@b.com" --password="password"
drush user-add-role "administrator" newuser

2.13.5. Module

drush vset maintenance_mode 1 set to maintenance mode drush vset maintenance_mode 0 remove maintenance mode drush dl devel Download a module drush en devel Enable a module drush dis devel Disable a module drush pmu devel Uninstall a module. Won't delete code files drush up <modulename> -y Update a module. Don't need to visit update.php drush up --no-core Update all modules. drush pml List all modules that are enabled, disabled, not installed, installed. drush pml --pipe list modules in code name 

drush up <modulename> = drush upc <modulename> + drush updb

Remove incorrectly removed modules in db d7:ts:module is missing

2.13.6. Detect if it's drush

if (drupal_is_cli() && function_exists('drush_main')) return true;
return false;

2.13.7. Watchdog

2.13.7.1. Delete
;; delete all logs
drush wd-del all

drush wd-del --type=cron

drush wd-del --severity=notice
2.13.7.2. Tail Recent Log Messages
drush watchdog-show --tail --full --count=50

2.13.8. SQL Query

drush sql-query "SELECT * FROM users WHERE uid=1"

2.13.9. Drupal Cache

Clear all Drupal Cache 

drush cc all

2.13.10. Fields

// Field Name, Feild Type, Bundles drush field-info fields

// Field type, Default widget, Widgets drush field-info types

2.13.11. Custom Drush

hook_drush_command() and drush_hook_your_command() in hook.drush.inc 

function li_drush_command() {
  $items['feeds-import-tnt']=[
    'description' => '...',
    'options' => [
      'feed-id' => ['description' => dt('Feed ID to import') ]
    ], // if there is no param, don't include 'options'
    'examples' => [
      'drush feeds-import-tnt --feed-id=123' => "Test example",
    ]
  ];
  return $items;
}

function drush_li_feeds_import_tnt() {
  $feed_id = drush_get_option('feed-id');
}
// drush feeds-import-tnt --feed-id=123

drupal_set_message()

  • stdout. No logging. Can be used multiple times in a drush command.
  • [status] at the end of each line
  • not print out immediately

Use the following every 3 minutes to prevent drush command timeout 

print "hello\n"; // print out immediately
drush_print(); // drush_print uses print
drush_print_r($array);

2.13.12. Pantheon Drush

2.14. Node API d7:api:node

https://api.drupal.org/api/drupal/modules%21node%21node.api.php/7.x https://api.drupal.org/api/drupal/modules%21node%21node.api.php/group/node_api_hooks/7.x

Node object

$node->nid :: node id $node->type

2.14.1. Update a node

$node = node_load($nid);
$node->fieldname['und'][0]['value'] = 'field value';
node_save($node);

d7:field_attach_update doens't trigger hook_node_presave and other node hooks 

$article_nodes = node_load_multiple(array(), array('type' => 'article'));
foreach ($article_nodes as $article_node) {
  $article_node->body[LANGUAGE_NONE][0]['value'] = 'body value';
  field_attach_update('node', $article_node);
}

Recommended: use d7:entity_metadata_wrapper to update to avoid putting language code

2.15. Views d7:viewsapi

2.15.1. UI

2.15.1.1. Contextual filter with term name not term id

Choose Content: Has taxonomy term ID Check Specify validation criteria Validator Taxonomy Term, Filter value type Term name converted to Term ID Check Transform dashes in URL to spaces in term name filter values

2.15.1.2. Path vs Link

Content: Path with Rewrite Result option "Use absolute link" is the raw path.

2.15.1.3. Field value is taxonomy ID not taxonomy name

Use Rewrite Result "[field_name-tid]"

2.15.1.4. Row Class

Use token [field_my_field_name] in row class. For taxonomy term to show id, you will have to create another field [field_my_field_name_1] shown as plain text and don't create label and rewrite the field to show Raw id and then reference [field_my_field_name_1] in row class

2.15.1.5. Sort Tax Term Field

node has a field called Category which is Tax Term Cat123 In Views, add relationship Content:Category (the node field) and under SORT CRITERIA in View setting, add Taxonomy Term either Weight or Name and select the Taxonomy term Cat123.

2.15.2. Entity Reference View

Require d7:entityreference module d7:entity reference view

2.15.3. View Object

Hooks use $view object.

2.15.3.1. Properties

args :: array( 0 => '37', ) name :: 'tt_taxonomy' current_display :: 'page'

2.15.3.2. Methods

$v->get_items_per_page(); / Return nothong in hook_views_pre_build $v->set_items_per_page(10); / hook_views_pre_build

2.15.4. hook_views_api

Views 3.x
/**
 * Implements hook_views_api().
 */
function my_module_views_api() {
  return array(
    'api' => 3,
  );
}

All Views hook code should be placed in my_module.views.inc which is located in module root directory if 'path' is not defined. All hooks' callbacks (Class definition) such as plugin and custom filter callback should be included in my_module.info as separate files

2.15.5. hook_views_query_alter

Put it in MODULENAME.views.inc based on hook_views_api. But I had success to just include this hook in .module file One way to avoid setup the *.views.inc is to use Views GUI to assign an Query Tag then use hook_query_TAG_alter but it runs twice.. So stick with hook_views_query_alter 

// dpm($query->where);
// One condition
$c1 = [
 'field' => 'node.status',
 'value' => 1,
 'operator' => '='
];

$c2 = [
 'field' => 'node_search_index.word',
 'value' => '% burger %',
 'operator' => 'LIKE'
];

$c3 = [
 'field' => 'node.uid = :node_uid',
 'value' => [':node_uid' => '5'],
 'operator' => 'formula'
];

$c4 = [
 'field' => 'node.type',
 'value' => ['ns_newsletter'],
 'operator' => 'in'
];

// a condition group
$cg1 = [
  'field' => object::Drupal\Core\Database\Query\Condition
];

[ 'conditions' => [ $c1, $c2 ]
  'args' => [],
  'type' => 'AND'
],
[ 'conditions' => [ $c3, $c4 ]
  'args' => [],
  'type' => 'AND'
],
[ 'conditions' => [ $cg1 ]
  'args' => [],
  'type' => 'AND'
]

function mymod_views_query_alter(&$view, &$query) {
  dpm($view, __FUNCTION__);
  dpm($query, __FUNCTION__);
  if ($view->name == 'lili_view' && $view->current_display == 'page') {
    foreach ($query->where as &$condition_group) {
      _recursively_alter_query_conditions($condition_group['conditions']);
    }
  }
  // add a where condition group
  // views_plugin_query_default::add_where($group, $field, $value = NULL, $operator = NULL)
  $cg_count = count($query->where);
  $query->add_where($cg_count, '0', '1', '=');
}

// helper function: (takes in conditions group argument)
function _recursively_alter_query_conditions(&$conditions) {
  // foreach condition in condition group
  foreach ($conditions as &$condition) {
    // if condition is itself a condition group
    if (isset($condition['field']) && is_a($condition['field'], 'Drupal\Core\Database\Query\Condition')) {
      // call the helper function on it
      _recursively_alter_query_conditions($condition['field']->conditions());
    }
    else {
      // check if we want to alter the condition and if so alter it
      _alter_query_condition($condition);
    }
  }
}

// separate helper function to determine if the condition is one we want to alter
function _alter_query_condition(&$condition) {
  if (isset($condition['field']) && ($condition['field'] === 'node_search_index.word')) {
    $condition['value'] = "%{$condition['value']}%";
    $condition['operator'] = 'LIKE';
  }
  if (isset($condition['field']) && ($condition['field'] === 'node_search_dataset.data')) {
    // here we're using trim to eliminate both the unwanted whitespace and the '%' symbols,
    // which then get added back via string concatenation
    $condition['value'] = "%" . trim($condition['value'], " \t\n\r\0\x0B%") . "%";
  }
}

2.15.6. hook_views_pre_build

2.15.7. hook_views_pre_render

2.15.7.1. Change Global Custom Text

Change the field value with tokens in Views UI 

function lili_views_pre_render(&$view) {
        if ($view->name == 'lili_view' && $view->current_display == 'page') {

                // Change value for specific rows
                foreach ($view->result as $k => $r) {
                        if ($f = &$r->field_field_flag) {
                                $o = $f[0]['rendered']['#markup'];
                                $o = theme('lili_theme_flair', array('text'=>$o));
                                $f[0]['rendered']['#markup'] = $o;
                        }

                        // Taxonomy field
                        if ($f = &$r->field_field_category) {
                                $exclude_tids = array(1,2,3);
                                foreach ($f as $k => $v) {
                                        if (!in_array($v['raw']['tid'], $exclude_tids)) {
                                                // Remove from display
                                                unset($f[$k]);
                                        }
                                }
                        }

                }

                // Only change global custom text for a subset of contextual filter value
                // The first context filter value
                if ($view->args[0] == 'filter_value') {
                        // Check View UI Theme Information to get the Global Custom Text field ID
                        if (isset($view->field['field_thumbnail']->options['exclude'])) {
                                // Exclude display
                                $view->field['field_thumbnail']->options['exclude'] = 1;
                        }
                        if (isset($view->field['nothing']->options['alter']['text'])) {
                                // Change a field's CSS Class setting for all records
                                $view->field['nothing']->options['element_class'] = '';
                                $o = $view->field['nothing']->options['alter']['text'];
                                $view->field['nothing']->options['alter']['text'] = 'hello'.$o;
                        }
                }
        }
}

2.15.8. Custom View Filter hook_views_data_alter

http://precessionmedia.com/blog/how-create-custom-filter-handler-views 

/**
 * Implements hook_views_data_alter().
 */
function my_module_views_data_alter(&$data) {
  $data['node']['title_count']['title'] = 'Title word count';
  $data['node']['title_count']['help'] = 'Count the number of words in titles.';
  $data['node']['title_count']['filter']['handler'] = 'my_module_handler_filter_field_count';
}

my_module.info 

files[] = my_module_handler_filter_field_count.inc

my_module_handler_filter_field_count.inc 

class my_module_handler_filter_field_count extends views_handler_filter_numeric {
  function operators() {
    $operators = parent::operators();
    // We won't be using regex in our example
    unset($operators['regular_expression']);

    return $operators;
  }

  // Helper function to return a sql expression
  // for counting words in a field.
  function field_count() {
    // Set the real field to the title of the node
    $this->real_field = 'title';

    $field = "$this->table_alias.$this->real_field";
    return "LENGTH($field)-LENGTH(REPLACE($field,' ',''))+1";
  }

  // Override the op_between function
  // adding our field count function as parameter
  function op_between($field) {
    $field_count = $this->field_count();

    $min = $this->value['min'];
    $max = $this->value['max'];

    if ($this->operator == 'between') {
      $this->query->add_where_expression($this->options['group'], "$field_count BETWEEN $min AND $max");
    }
    else {
      $this->query->add_where_expression($this->options['group'], "($field_count <= $min) OR ($field_count >= $max)");
    }
  }

  // Override the op_simple function
  // adding our field count function as parameter
  function op_simple($field) {
    $field_count = $this->field_count();

    $value = $this->value['value'];

    $this->query->add_where_expression($this->options['group'], "$field_count $this->operator $value");
  }
}

In Views UI, add a filter with name "Title word count" with descript "Count the number of words in titles."

2.15.9. Better Exposed Filters Module d7:better exposed filters

If you want checkboxes or radio buttons instead of the Views Basic Exposed Form which is dropdown select.

Change BEF form hook_form_views_exposed_form_alter 

function lili_form_views_exposed_form_alter(&$form, &$form_state) {
  if (isset($form['#id'])) {
    switch ($form['#id']) {
      case 'views-exposed-form-your-form-name':
        // ...
        break;
    }
  }
}

The form id is views-exposed-form-view–machine-name-display-machine-name

2.15.10. Custom Views Plugin hook_views_plugins

2.15.11. Views API Order

hook_views_pre_view hook_views_pre_build hook_views_post_build hook_views_pre_execute hook_views_post_execute hook_views_pre_render hook_views_post_render

2.15.12. Embed View and Display

views_embed_view('view_name','display_id'); Doesn't display view title 

$arg1 = $arg2 = '123';
print views_embed_view('view_name','display_id',$arg1,$arg2);

$my_view_name = 'magazines';
$my_display_name = 'block_1';
$my_view = views_get_view($my_view_name);
if (is_object($my_view)) {
  $my_view->set_display($my_display_name);
  $my_view->pre_execute();
  echo $my_view->render($my_display_name);
}

2.15.13. View Theming d7:View_Theming

View, named foobar. Style: unformatted. Row styel: Fields. Display: Page.

  • views-view-foobar–page.tpl.php
  • views-view-page.tpl.php
  • views-view–foobar.tpl.php
  • views-view.tpl.php
  • views-view-unformatted–foobar–page.tpl.php
  • views-view-unformatted–page.tpl.php
  • views-view-unformatted–foobar.tpl.php
  • views-view-unformatted.tpl.php
  • views-view-fields–foobar–page.tpl.php
  • views-view-fields–page.tpl.php
  • views-view-fields–foorbar.tpl.php
  • views-view-fields.tpl.php

2.16. Ellipsis d7:ellipsis

Use truncate_utf8 for plain text string 

truncate_utf8($string, $max_length, $wordsafe = FALSE, $add_ellipsis = FALSE, $min_wordsafe_length = 1)

This can be used in custom module. Trim first, then remove scraps of html e.g. <a>fda</, <a>fda</a and <a>fda< will become <a>fda 'html' => true will close any unclosed html tags. 

views_trim_text(array(
            'max_length' => 200,
            'word_boundary' => TRUE,
            'ellipsis' => TRUE,
            'html' => TRUE, // optional
          ), 'Long String');
// Output
Long String<span class="ellipsis">...</span>

Source code 

function views_trim_text($alter, $value) {
  if (drupal_strlen($value) > $alter['max_length']) {
    $value = drupal_substr($value, 0, $alter['max_length']);
    // TODO: replace this with cleanstring of ctools
    if (!empty($alter['word_boundary'])) {
  $regex = "(.*)\b.+";
  if (function_exists('mb_ereg')) {
    mb_regex_encoding('UTF-8');
    $found = mb_ereg($regex, $value, $matches);
  }
  else {
    $found = preg_match("/$regex/us", $value, $matches);
  }
  if ($found) {
    $value = $matches[1];
  }
    }
    // Remove scraps of HTML entities from the end of a strings
    $value = rtrim(preg_replace('/(?:<(?!.+>)|&(?!.+;)).*$/us', '', $value));

    if (!empty($alter['ellipsis'])) {
  $value .= t('...');
    }
  }
  if (!empty($alter['html'])) {
    $value = _filter_htmlcorrector($value);
  }

  return $value;
}

function _filter_htmlcorrector($text) {
  return filter_dom_serialize(filter_dom_load($text));
}
function filter_dom_load($text) {
  $dom_document = new DOMDocument();
  // Ignore warnings during HTML soup loading.
  @$dom_document->loadHTML('<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8" /></head><body>' . $text . '</body></html>');

  return $dom_document;
}

function filter_dom_serialize($dom_document) {
  $body_node = $dom_document->getElementsByTagName('body')->item(0);
  $body_content = '';

  if ($body_node !== NULL) {
    foreach ($body_node->getElementsByTagName('script') as $node) {
  filter_dom_serialize_escape_cdata_element($dom_document, $node);
    }

    foreach ($body_node->getElementsByTagName('style') as $node) {
  filter_dom_serialize_escape_cdata_element($dom_document, $node, '/*', '*/');
    }

    foreach ($body_node->childNodes as $child_node) {
  $body_content .= $dom_document->saveXML($child_node);
    }
    return preg_replace('|<([^> ]*)/>|i', '<$1 />', $body_content);
  }
  else {
    return $body_content;
  }
}

// Adds comments around the <!CDATA section in a dom element.
function filter_dom_serialize_escape_cdata_element($dom_document, $dom_element, $comment_start = '//', $comment_end = '') {
  foreach ($dom_element->childNodes as $node) {
    if (get_class($node) == 'DOMCdataSection') {
  // See drupal_get_js().  This code is more or less duplicated there.
  $embed_prefix = "\n<!--{$comment_start}--><![CDATA[{$comment_start} ><!--{$comment_end}\n";
  $embed_suffix = "\n{$comment_start}--><!]]>{$comment_end}\n";

  // Prevent invalid cdata escaping as this would throw a DOM error.
  // This is the same behavior as found in libxml2.
  // Related W3C standard: http://www.w3.org/TR/REC-xml/#dt-cdsection
  // Fix explanation: http://en.wikipedia.org/wiki/CDATA#Nesting
  $data = str_replace(']]>', ']]]]><![CDATA[>', $node->data);

  $fragment = $dom_document->createDocumentFragment();
  $fragment->appendXML($embed_prefix . $data . $embed_suffix);
  $dom_element->appendChild($fragment);
  $dom_element->removeChild($node);
    }
  }
}

2.17. Form

2.17.2. Form alter hooks

Called in order, from general to specific

hook_form_alter
for all forms
hook_form_BASE_FORM_ID_alter
Refer BASE_FORM_ID to $form_state['build_info']['base_form_id']
hook_form_FORM_ID_alter
Refer FORM_ID to $form['#id']

Set default value and hide a form from display or filling 

// Entity reference form field
$form['field_dealer'][LANGUAGE_NONE][0]['target_id']['#default_value'] = 123;
$form['field_dealer'][LANGUAGE_NONE][0]['#printed'] = TRUE;

hook_form_FORM_ID_alter(&$form, &$form_state, $form_id)

2.17.3. Form IDs of system forms

Form

node
node new/edit
$form['']
$form['actions']['submit']['#submit'] = array('node_form_submit')
user_profile
page/user/<uid>

Alter

lili_node_form_alter
hook_node_form_alter
node type (content type) ad_listing and alter its node new/edit form
lili_form_ad_listing_node_form_alter using hook_form_FORM_ID_alter hook where FORM_ID is ad_listing_node_form

Some modules alter the form before your custom alters. Such as field dependency module d7:module:conditional fields

2.17.4. Confirm Form

e.g. Confirm before delete 

// Menu
$items['dealer_user/%/delete'] = array(
        'title' => 'Delete a user',
        'page callback' => 'drupal_get_form',
        'page arguments' => array('lili_delete_dealer_user_confirm',1),
        'access callback' => 'lili_user_has_delete_right',
        'access arguments' => array(1, array('Dealer Admin')),
        'type' => MENU_LOCAL_TASK,
);

function lili_delete_dealer_user_confirm($form, &$form_state, $id) {
        $form['delete'] = array(
                '#type' => 'value',
                '#value' => $id,
        );
        return confirm_form($form,
                t('Are you sure you want to delete this user?'), // Question
                'dealer_manage_users',  // Path to go to if No
                t('This action cannot be undone.'), // Description
                t('Delete Button'), // Yes: text
                t('Cancel Button') // No: text
                //, $name default 'confirm': internal name used to refer to the confirmation item.
        );
}

function lili_delete_dealer_user_confirm_submit($form, &$form_state) {
        if ($uid = $form_state['values']['delete']) {
                $u = user_load($uid);
                user_cancel();
                drupal_set_message(t('The user account has been deleted!'));
        }
        $form_state['redirect'] = 'dealer_manager_users';
}

function lili_user_has_delete_right($uid = 0, array $roles) {
        global $user;
        // If user is anonymous
        if (user_is_anonymous()) {
                return FALSE;
        }
        if (in_array('administrator', $user->roles)) {
                return TRUE;
        }
        if (array_intersect($roles, $user->roles)) {
                return TRUE;
        }
}

2.17.5. form_load_include

Include files whenever a form is loaded. form_load_include(&$form_state, $type, $module, $name = NULL) {} / if name is omitted, then $module.$type is loaded. / return The filepath of the loaded include file, or FALSE if the include file was / not found or has been loaded already. / file is always included even the form is cached. form_load_include($form_state, 'inc', 'tnt', 'tnt.pages');

2.17.6. Form Ajax

https://www.drupal.org/docs/7/api/javascript-api/ajax-forms-in-drupal-7

https://api.drupal.org/api/drupal/developer%21topics%21forms_api_reference.html/7.x#ajax 

/**
 * Ajax-enabled select element causes replacement of a set of checkboxes
 * based on the selection.
 */
function ajax_example_autocheckboxes($form, &$form_state) {

  $default = !empty($form_state['values']['howmany_select']) ? $form_state['values']['howmany_select'] : 1;

  $form['howmany_select'] = array(
    '#title' => t('How many checkboxes do you want?'),
    '#type' => 'select',
    '#options' => array(1 => 1, 2 => 2, 3 => 3, 4 => 4),
    '#default_value' => $default,
    '#ajax' => array(
      'callback' => 'ajax_example_autocheckboxes_callback',
      'wrapper' => 'checkboxes-div',
      'method' => 'replace',
      'effect' => 'fade',
    ),

  );


  $form['checkboxes_fieldset'] = array(
    '#title' => t("Generated Checkboxes"),
    // The prefix/suffix provide the div that we're replacing, named by
    // #ajax['wrapper'] above.
    '#prefix' => '<div id="checkboxes-div">',
    '#suffix' => '</div>',
    '#type' => 'fieldset',
    '#description' => t('This is where we get automatically generated checkboxes'),
  );

  $num_checkboxes = !empty($form_state['values']['howmany_select']) ? $form_state['values']['howmany_select'] : 1;
  for ($i = 1; $i <= $num_checkboxes; $i++) {
    $form['checkboxes_fieldset']["checkbox$i"] = array(
      '#type' => 'checkbox',
      '#title' => "Checkbox $i",
    );
  }

  $form['submit'] = array(
    '#type' => 'submit',
    '#value' => t('Submit'),
  );

  return $form;
}

/**
 * Callback element needs only select the portion of the form to be updated.
 * Since #ajax['callback'] return can be HTML or a renderable array (or an
 * array of commands), we can just return a piece of the form.
 */
function ajax_example_autocheckboxes_callback($form, $form_state) {
  return $form['checkboxes_fieldset'];
}
#ajax['event']
which jquery event triggers AJAX. Don't need to set. Default is fine.
#ajax['callback']
return HTML or renderable array to replace #ajax['wrapper']. By the time it gets called, the form is recalculated/rebuilt in PHP and you can return $form['the_field_name'] directly. The callback function can't change any state or form settings. Do it in form builder functions: hook_form, hook_form_alter. Callback function is called after all the form processing fuctions.
#ajax['wrapper']
HTML container.

2.17.7. Form field loaded with terms of a taxonomy fields d7:form:taxonomy field

$parts_makes = ['0' => t('Make')]; // First option
$_terms = taxonomy_allowed_values(field_info_field('field_parts_make'));

// [123] => term123_name, [345] => term345_name, ...

if ($language->language == 'en') {
  $parts_makes += $_terms;
}
else {
  $parts_make_terms = $_terms;
  foreach ($parts_make_terms as $tid => $name) {
    $i18n_object = i18n_get_object('taxonomy_term', $tid);
    if ($translated_term = $i18n_object->localize($language->language)) {
      $parts_makes[$tid] = (!empty($translated_term->name)) ? $translated_term->name : $parts_make_terms[$tid];
    }
  }
}

$form['make'] = [
  '#type' => 'select',
  '#options' => $parts_makes,
];

2.18. Action

hook_info_action return ['action_name' => $an_action];

$an_action :: array

'permissions'
this is specific for View Bulk Operations (VBO). e.g. array('switch users')
'behavior'
array. VBO uses one of views_property, changes_property, creates_property, deletes_property

VBO Permissions

2.19. Field API d7:api:field

When a field associates with an entity, a widget can be specified A field with an entity can have one formatter for each view mode.

2.19.1. hook_field_is_empty d7:api:field:hook_field_is_empty

modules/field/field.api.php hook_field_is_empty($item, $field) /modules/field/modules[fieldtype]/[fieldtype].module Or any modules that implements hook_field_is_empty

Core fieldtype:

  • field_sql_storage
  • list
  • number
  • options
  • text
  • taxonomy
  • image
  • file
$info = field_info_field($field_name);
$function = $info['module'] . '_field_is_empty';

if (function_exists($function)) {
  $value = field_get_items('node', $node, $field_name);
  $is_empty = $function($value, $info);
}

2.19.2. Field Formatter d7:api:field:formatter

Content Type > Manage Display > Format for a field hook_field_formatter_info tells Drupal a new field formatter that will apply to what field types. If a formatter has a setting, a summary and a form needs to be defined in order to show a gear icon next to summary in Manage Display

hook_field_formatter_info :: hook_field_formatter_info_alter 

function text_field_formatter_info() {
  return array(
    'text_default' => array(
      'label' => t('Default'),
      'field types' => array('text', 'text_long', 'text_with_summary'),
    ),
    'text_plain' => array(
      'label' => t('Plain text'),
      'field types' => array('text', 'text_long', 'text_with_summary'),
    ),

    'text_trimmed' => array(
      'label' => t('Trimmed'),
      'field types' => array('text', 'text_long', 'text_with_summary'),
      'settings' => array('trim_length' => 600),
    ),

    'text_summary_or_trimmed' => array(
      'label' => t('Summary or trimmed'),
      'field types' => array('text_with_summary'),
      'settings' => array('trim_length' => 600),
    ),
  );
}

hook_field_formatter_settings_summary 

function text_field_formatter_settings_summary($field, $instance, $view_mode) {
  $display = $instance['display'][$view_mode];
  $settings = $display['settings'];

  $summary = '';

  if (strpos($display['type'], '_trimmed') !== FALSE) {
    $summary = t('Trimmed limit: @trim_length characters', array('@trim_length' => $settings['trim_length']));
  }

  return $summary;
}

hook_field_formatter_settings_form 

function text_field_formatter_settings_form($field, $instance, $view_mode, $form, &$form_state) {
  $display = $instance['display'][$view_mode];
  $settings = $display['settings'];

  $element = array();

  if (strpos($display['type'], '_trimmed') !== FALSE) {
    $element['trim_length'] = array(
      '#title' => t('Trimmed limit'),
      '#type' => 'textfield',
      '#field_suffix' => t('characters'),
      '#size' => 10,
      '#default_value' => $settings['trim_length'],
      '#element_validate' => array('element_validate_integer_positive'),
      '#description' => t('If the summary is not set, the trimmed %label field will be shorter than this character limit.', array('%label' => $instance['label'])),
      '#required' => TRUE,
    );
  }

  return $element;
}

hook_field_formatter_view :: use settings to return result (HTML). Use hook_field_formatter_prepare_view to add info for field values being displayed. 

function text_field_formatter_view($entity_type, $entity, $field, $instance, $langcode, $items, $display) {
  $element = array();

  switch ($display['type']) {
    case 'text_default':
    case 'text_trimmed':
      foreach ($items as $delta => $item) {
        $output = _text_sanitize($instance, $langcode, $item, 'value');
        if ($display['type'] == 'text_trimmed') {
          $output = text_summary($output, $instance['settings']['text_processing'] ? $item['format'] : NULL, $display['settings']['trim_length']);
        }
        $element[$delta] = array('#markup' => $output);
      }
      break;

    case 'text_summary_or_trimmed':
      foreach ($items as $delta => $item) {
        if (!empty($item['summary'])) {
          $output = _text_sanitize($instance, $langcode, $item, 'summary');
        }
        else {
          $output = _text_sanitize($instance, $langcode, $item, 'value');
          $output = text_summary($output, $instance['settings']['text_processing'] ? $item['format'] : NULL, $display['settings']['trim_length']);
        }
        $element[$delta] = array('#markup' => $output);
      }
      break;

    case 'text_plain':
      foreach ($items as $delta => $item) {
        $element[$delta] = array('#markup' => strip_tags($item['value']));
      }
      break;
  }

  return $element;
}

Usage 

$node = node_load($view->result[0]->_field_data['nid']['entity']->nid);
$settings = array(
  'label'    => 'hidden',
  'type'     => 'text_summary_or_trimmed',
  'settings' => array('trim_length' => 800),
);
$node_summary = field_view_field('node', $node, 'body', $settings);
print render($node_summary);

2.19.3. field_get_items d7:field_get_items

Return the field items in the language they currently would be displayed. $items = field_get_items($entity_type, $entity, $field_name, $langcode = NULL)

If there is no field associated with $entity or the field is not a field belongs to $entity, it will return false. 

// field_get_items($entity_type, $entity, $field_name, $langcode = NULL)
// field_get_items('node', $node, 'field_my_field_name' )

if (!empty(field_get_items('node', $node, 'field_my_field_name'))) {
    // field has at least one value
}

// or use this to determine how many values a field has
function hasValues($field_name,$entity, $entity_type='node', $langcode = NULL) {
    $r     = [
  'has'   => FALSE,
  'count' => 0
    ];
    $value = field_get_items($entity_type, $entity, $field_name, $langcode);
    if (is_array($value)) {
  $r = [
    'has' => true,
    'count' => count($value)
  ];
    }
    return $r;
}

To check if a field value is empty d7:api:field:hook_field_is_empty

After checking it has values, then you can render($content['field_name']) in node.tpl.php

2.19.3.1. Image field

fid => "768" uid => "1" filename => "splash-photo.jpg" uri => "public://splash-photo.jpg" filemime => "image/jpeg" filesize => "129265" status => "1" timestamp => "1518792000" alt => "" title=> "" width => "727" height => "426"

2.20. Database d7:database

2.20.1. Close Comment for Existing Nodes

0 = No, 1 = Closed (read only), 2 = Open (Read/Write) 

UPDATE node, node_revision
SET node.comment = 1, node_revision.comment = 1
WHERE node_revision.nid = node AND node.type = 'article'

2.20.2. Structure

2.20.2.1. node
nid PK
int(10) unsigned
vid
The current node_revision.vid version id
uid
int(11) d:0 user/author id
type
varchar(32) d:'' e.g. article
title
varchar(255) d:''
created
int(11) d:0 unix timestamp
changed
int(11) d:0 unix timestamp
comment
int(11) d:0 2 = open (read/write), 1 = closed (read only), 0 = no
sticky
int(11) d:0 Boolean to display node at the top of query result
promote
int(11) d:0 Boolean to show on homepage
tnid
int(10) unsigned d:0 translation set id
translate
int(11) d:0 Boolean to mark translation is needed
2.20.2.2. field_config
id PK
int 11
field_name
vchar 32 e.g. field_mileage_units
type
vchar 128
  • text
  • datetime
  • link_field
  • phone
  • list_text
  • list_boolean
  • file
  • taxonomy_term_reference
  • entityreference
module
vchar 128
  • text
  • date
  • link
  • phone
  • list
  • file
  • taxonomy
  • entityreference
active
tinyint 4. bool NOT NULL
storage_type
vchar 128 e.g. field_sql_storage
storage_module
vchar 128 e.g. field_sql_storage
locked
tinyint 4 don't know..
data
longblob NOT NULL serialized data containing the field properties
cardinality
tinyint 4 default 0 don't know..
translatable
tinyint 4 default 0 don't know..
deleted
tinyint 4 default 0 don't know..
2.20.2.3. field_data_body
  • Common fields
    entity_type PK
    varchar(128) d:'' this field is attached to e.g. node
    entity_id PK
    int(10) unsigned e.g. node id
    bundle
    varchar(128) d:'' e.g. node type article
    deleted PK
    tinyint(4) d:0 not deleted
    language PK
    varchar(32) d:'' e.g. und
    delta PK
    int(10) unsigned used for ulti-value fields, sequence number, start as 0
    revision_id PK
    int(10) unsigned
  • Unique fields
    body_value
    longtext
    body_summary
    longtext
2.20.2.4. field_data_[field_my_text_or_longtext]
  • Text
    field_data_[field_my_text]_value
    varchar(255)
    field_data_[field_my_text]_format
    varchar(255)
  • Longtext
  • longtext
  • varchar(255)
2.20.2.5. field_data_[field_my_date_unix_timestamp]
field_data_[field_my_date_unix_timestamp]_value
int(11)
2.20.2.6. field_data_[field_custom_term_reference]
field_custom_term_referecen_tid
int
2.20.2.7. field_data_field_mileage_units list_text
entity_type PK I-A
vchar 128 default '' NOT NULL The entity type this data is attached to
  • node
(no term)
bundle I-A:: vchar 128 default '' NOT NULL The field instance bundle to which this row belongs, used when deleting a field instance
  • my_node_type
deleted PK I-A
tinyint 4 default 0 NOT NULL
entity_id PK I-A
int 10 NOT NULL the entity id this data is attached to.
revision_id I-A
int 10 or NULL if the entity type is not versioned
language PK I-A
vchar 32 default '' NOT NULL The language for this data item
delta PK
int 10 NOT NULL the sequence number for this data item, used for multi-value fields
field_mileage_units_value I-A
vchar 255 NULL
2.20.2.8. field_data_field_contributor entity reference
field_contributor_target_id
int
2.20.2.9. taxonomy_term_data
tid*
int
vid
int, vocabulary id. see taxonomy_vocabulary
name
string, term name
weight
int
2.20.2.10. taxonomy_vocabular

vid* :: name :: machine_name

2.20.2.11. url_alias

pid* :: int, path id source :: string, e.g. node/2, taxonomy/term/11 alias :: string language :: string, e.g. und

2.20.2.12. Export Query
SELECT n.title,
  b.body_value,
  GROUP_CONCAT(t.name SEPARATOR ', ') AS Category,
  IFNULL(na.title,'') AS author,
  CONCAT('http://domain.com/',u.alias) as url,
  FROM_UNIXTIME(n.created) as post_date

FROM node AS n
  INNER JOIN field_data_body b ON b.entity_id=n.nid AND b.delta=0
  INNER JOIN field_data_field_category c ON c.entity_id=n.nid AND c.deleted=0
  LEFT JOIN taxonomy_term_data t ON t.tid = c.field_category_tid
  INNER JOIN field_data_field_contributor a ON a.entity_id=n.nid AND a.deleted=0
  LEFT JOIN node as na ON na.nid=a.field_contributor_target_id
  INNER JOIN url_alias u ON u.source = concat('node/',n.nid)

WHERE FROM_UNIXTIME(n.created)
      BETWEEN '2014-01-01 00:00:00' AND '2014-12-31 23:59:59'
  AND n.type ='article'
GROUP BY n.nid
ORDER BY n.created DESC

Export taxonomy vocabular 

SELECT
  t.tid                                         AS tid,
  t.name,
  h.parent                                      AS parent_tid,
  t.weight,
  CONCAT('http://yourweb.com/', u.alias)                        AS url

FROM taxonomy_vocabulary v
  LEFT JOIN taxonomy_term_data t ON t.vid = v.vid
  LEFT JOIN taxonomy_term_hierarchy h ON h.tid = t.tid
  LEFT JOIN field_data_field_mega_title mega_title
    ON mega_title.entity_id = t.tid
  LEFT JOIN field_data_field_mega_story_title mega_story_title
    ON mega_story_title.entity_id = t.tid
  LEFT JOIN field_data_field_landing_page_type splash
    ON splash.entity_id = t.tid
  LEFT JOIN url_alias u ON u.source = concat('taxonomy/term/', t.tid)
WHERE
  0 = 0
  AND v.machine_name = 'articles'

ORDER BY h.parent, t.weight
2.20.2.13. Export File URI, file_managed d7:db:file uri
SELECT n.nid, n.title
,ds_i_fm.uri AS splash_image -- e.g. public://abc.jpg
,REGEXP_REPLACE(ds_i_fm.uri, "^(s3://)(.*)", 'http://s3.amazonaws.com/abc.com/\\2') AS splash_image, -- s3://abc.jpg
FROM node AS n
LEFT JOIN field_data_field_image ds_i ON ds_i.entity_id=n.nid AND ds_i.deleted=0
LEFT JOIN file_managed ds_i_fm ON ds_i_fm.fid = ds_i.field_image_fid
WHERE n.type = 'article'
AND n.nid = 123
2.20.2.14. file_usage

fid id :: e.g. node id

2.20.2.15. system

d7:ts:module is missing

2.20.2.16. users

*uid :: int(10) name :: pass :: mail :: email init :: initial email created :: int(11), unix time access :: int(11), unix time. previous time user accessed the site login :: int(11), unix time status :: tinyint(4)

2.20.2.17. users_role

uid rid

2.20.2.18. role

rid* name weight

2.21. Query

Use db_query as much as possible. db_select is slow because it calls alter hooks. Both can only be looped once using foreach

Use db_query or db_select to get all nids, load all nodes at once and then manipulate fields.

db_query is meant for simple select query. db_select is dynamic query

2.21.1. db_select

$q = db_select( 'node', 'n' );
$q->join( 'field_data_body', 'b', 'n.nid=b.entity_id AND b.delta=0' );
$q->innerJoin( 'field_data_field_category', 'nc'
  , 'n.nid=nc.entity_id AND nc.deleted=0' );
$q->innerJoin( 'taxonomy_term_hierarchy', 'tH', 'tH.tid=nc.field_category_tid' );

if ( ! is_null( $termID ) ) {
  $articleCats = array( $termID );
}

$q->fields( 'n', array( 'nid', 'title', 'created' ) )
  ->fields( 'b', array( 'body_summary', 'body_value' ) )
  ->condition( 'n.status', 1 )// Published.
  ->condition( 'n.type', 'article' )// article not page.
  ->condition(
    db_or()
  ->condition( 'nc.field_category_tid', $articleCats, 'IN' )
  ->condition( 'tH.parent', $articleCats, 'IN' )
  )
  ->range( $i_start, $i_length )
  ->groupBy( 'n.nid' )
  ->orderBy( 'created', 'DESC' );

// debug a query
print_r($query->__toString());
print_r($query->arguments());

$r = $q->execute();

$nids = [ ];
foreach ( $result as $n ) {
  $nids[] = $n->nid;
}
$nodes = node_load_multiple( $nids );

foreach ( $nodes as $node ) {
  $nodeID    = $node->nid;
  $nodeTitle = $node->title;
  $nodeBody  = lili_getFieldSafe( $node, 'body' );
  $nodeURL   = url( "node/" . $node->nid );

  $nodeSplashObj = field_get_items( 'node', $node, 'field_splash_image' );

  if ( is_array( $nodeSplashObj ) ) {
    // $nodeSplashObj['#item']['uri'];
    // $nodeSplashObj['#item']['width'];
    // $nodeSplashObj['#item']['height'];
  }

  // Entity Reference
  $nodeFlag = field_get_items('node',$nodeObj,'field_flag');
  if ($nodeFlag) {
    $nodeFlag = field_view_value('node', $node, 'field_flag', $nodeFlag[0]);
    $nodeFlag = $nodeFlag['#markup'];
    $nodeFlag = theme_infscroll_flair(array('text'=>$nodeFlag));
  }
  else {
    $nodeFlag = '';
  }

}

function lili_getFieldSafe( $node, $field ) {
  $s = field_get_items( 'node', $node, $field );
  if ( $s ) {
    $s = ( $s[0]['safe_value'] ) ? $s[0]['safe_value'] : '';
  } else {
    $s = '';
  }

  return $s;
}

2.21.2. db_query

https://www.drupal.org/docs/7/api/database-api/static-queries https://api.drupal.org/api/drupal/includes%21database%21database.inc/function/db_query/7.x 

$query = db_query("SELECT nid, title FROM {node}");
$records = $query->fetchAll(); // get all records into an indexed array of stdClass
$records = $query->rowCount();
foreach ($records as $record) {
  // Do something.
}

// placeholders should not be escaped or quoted
$result = db_query("SELECT nid, title FROM {node} WHERE created > :created", 
array(
  ':created' => REQUEST_TIME - 3600,
));

db_query("SELECT * FROM {node} WHERE nid IN (:nids)", array(':nids' => array(13, 42, 144)));

// query options, use 'target' and 'fetch' only
'target' => 'default', // or 'slave'
'fetch' => PDO::FETCH_OBJ // or PDO::FETCH_ASSOC, PDO::FETCH_NUM, PDO::FETCH_BOTH or a string of a class name

$sql = "SELECT name, quantity FROM goods WHERE vid = :vid";
$result = db_query($sql, array(':vid' => $vid));
if ($result) {
  while ($row = $result->fetchAssoc()) {
    // fetch next row as an associative array
    // vs fetch next row as an object $result->fetchObject()
    // Do something with:
    // $row['name']
    // $row['quantity']
  }
}

// Get the first column result set as one single array
$nids = $result->fetchCol();
// Column number can be specified otherwise default to first column
$nids = $result->fetchCol(2);

// say first column is node id, then load all nodes
$nodes = node_load_multiple( $nids );

Other fetch usage 

// Fetch data from specific column from next row
// Defaults to first column if not specified as argument
$data = $result->fetchColumn(1); // Grabs the title from the next row

// Retrieve all records as stdObjects into an associative array 
// keyed by the field in the result specified. 
// (in this example, the title of the node)
$result->fetchAllAssoc('title');

// Retrieve a 2-column result set as an associative array of field 1 => field 2.
$result->fetchAllKeyed();
// Also good to note that you can specify which two fields to use
// by specifying the column numbers for each field
$result->fetchAllKeyed(0,2); // would be nid => created
$result->fetchAllKeyed(1,0); // would be title => nid

2.21.3. db_delete

Same syntax as db_select 

$num_deleted = db_delete('node')
  ->condition('nid', 5)
  ->execute();

Clear cache is needed if field_data_field_xxx are deleted. Delete a required field for node (nid) d7:db:delete required field 

$q = db_delete('field_data_field_trailer_make');
$q->condition('entity_id', $node->nid);
$r = $q->execute();

2.21.4. hook_query_TAG_alter hook_query_TAG_alter

function mymod_query_node_is_not_tagged_alter(QueryAlterableInterface $query) {
  $query->leftJoin('field_data_field_tags', 'o', 'node.nid = o.entity_id AND o.entity_type = :entity_type');
  $query->isNull('o.field_tags_tid');
}

2.22. Entity

Node, user, taxonomy are all entities.

2.22.1. EntityFieldQuery d7:efq

Use this to get entities of one type that have certain:

  • entity properties (propertyCondition: status, changed, etc)
  • field values (fieldCondition)
  • entity meta data (entityCondition: bundle, entity_type, entity_id and revision_id)
  • It's essentially db_select except EFQ can't do joins
  • It's a lot slower
  • But it provides simple syntax
  • It's good for quick filtering entities by basic field values and return entity ids
  • Preferred over db_select
  • EFQ can return any type of entities including nodes of various types
  • EFQ can only return essential fields such as node nid, vid and type
  • Use EFQ result nids and node_load_multiple or entity_load
  • Use EFQ result nids and get values of a field for all nodes (load only one field for all nodes)
  • If you want custom fields and node->title, better to use node_load_multiple or entity_load
  • By default without addMetaData, EFQ requires permissions of all the fields!
$_q = new EntityFieldQuery();
$_r = $_q
->entityCondition('entity_type', 'node') // taxonomy_term, user
->entityCondition('bundle', 'newsletter') // node type or content type
->fieldCondition('field_active','value',1,'=') // boolean field, only 1 or 0
->fieldCondition('field_news_publishdate', 'value', $year . '%', 'like') // Text LIKE
->fieldOrderBy('field_ns_newsletter_unique_id', 'value', 'DESC') // Sort. refer to propertyOrderBy for created date
->fieldCondition('field_a_term_reference_field', 'tid', 123) // field Term Reference
->fieldCondition('field_a_term_reference_field', 'tid', $tids) // field Term Reference multiple has at least one value in array $tids
->fieldCondition('field_dealer', 'target_id', 12345, '=') // field Entity Reference, single value
->fieldCondition('field_contacts', 'target_id', 12345) // field Entity Reference, multiple values have one 12345
/* NOT IN
12345 not in mutliple target_id: you have to do one query to show all and do one with 'IN' and 
get the difference
$_r_all = (!empty($_r_all['node'])) ? $_r_all['node'] : array();
$_r_in_12345 = (!empty($_r_in_12345['node'])) ? $_r_in12345['node'] : array();
$_r = array_diff_key($_r_all, $_r_in_12345);
*/

->propertyCondition('status',1) 
// status, type, nid, uid, rui, type, created
// search for "Implements hook_entity_info()" will show all entity properties

// Operators
// Default '='
// <>, >, >=, <, <=
// STARTS_WITH, CONTAINS
// IN, NOT IN (does not work)
// BETWEEN :: propertyCondtion('created', array($start, $end), 'BETWEEN')

->propertyOrderBy('created', 'DESC') // sort by created date

->range(0,10)
// Random order. hook_query_TAG_alter
->addTag('random')

/* If you run EFQ in Drupal Cron, by default the user is anonymous
   You should run EFQ as a user
*/
->addMetaData('account', user_load(1))
->execute();

/* array(
 'node' => // follow the entity_type: e.g. node, taxonomy_term, user 
 array(
  28912 => obj(nid => 28912) // could be uid, nid, tid
  ...
 )
)
*/

// or $_r['user']
if (!empty($_r['node'])) {
  $nodes = node_load_multiple(array_keys($_r['node']));
  // If you just want to load a handful of fields, you don't have to load the whole nodes
  // $stories = $_r['node'];
  // Get all fields of the entity type first
  // $fields = field_info_instances('node', 'story'); // node type is story
  // Get the field id
  // $field_id = $fields['field_story_image']['field_id'];
  // Attach the field to the entities loaded by EFQ
  // field_attach_load('node', $stories, FIELD_LOAD_CURRRENT,
  //  array('field_id' => $field_id));
  // All the values of field_story_image of all nodes
  // $output = field_get_items('node', $stories, 'field_story_image');
}

2.22.2. Update a field of an entity only d7:field_attach_update

Recommend d7:entity_metadata_wrapper node_save could invoke other hooks and hence cause errors in saving a field value 

$node = node_load($nid);
$node->field_fieldname[LANGUAGE_NONE][0]['value'] = 'some value';
node_save($node);

Use field_attach_update() instead 

$node = node_load($nid);
$node->field_fieldname[LANGUAGE_NONE][0]['value'] = 'some value';

unset($node->field_unwanted_1); // remove unwanted fields
field_attach_update('node', $node);

Without node_load 

$node = new stdClass();
$node->id = $id_value; // node id
$node->type = $content_type; // aka content type

$node->field_fieldname[LANGUAGE_NONE][0]['value'] = 'some value';
field_attach_update('node', $node);

May be better to call field_attach_presave which node_save does 

field_attach_presave('node', $node); // call hook_field_attach_presave
field_attach_update('node', $node);

2.22.3. entity_metadata_wrapper d7:entity_metadata_wrapper

Get a field value of an entity (node)

  • Add a wrapper first $_wrapper=entity_metadata_wrapper('node',$node);
  • $node can be a node/entity object or the node/entity id.
  • Entity API module is needed.
  • List all fields using $_wrapper->getPropertyInfo();
  • Get a specific field $_wrapper->field_name->value(); or ->raw();
// Unified way of getting $node->title, $user->name, ...
$wrapper->label(); or $wrapper->title->value();

// Unified way of getting $node->nid, $user->uid, ...
$wrapper->getIdentifier();

// Unified way of getting $node->type, ...
$wrapper->getBundle();

// Text field
$old_value = $wrapper->field_my_text->value();
if (isset($old_value)) {
  // non-null, could be empty or 0
}
$wrapper->field_my_text = 'new value';

// Longtext
$wrapper->body->value(); // Array of: value, safe_value, format, and optionally summary + safe_summary.
$wrapper->body->value->value(); // Filtered value.
$wrapper->body->value->raw(); // Unfiltered value.
$wrapper->body->format->value(); // The selected text format.
$wrapper->body->value = 'new value';

// Link
$wrapper->field_my_link->value(); // Array of: url, title, attributes.
$wrapper->field_my_link->url->value(); // Hyperlink destination.
$wrapper->field_my_link->title->value(); // Hyperlink text.
$wrapper->field_my_link->attributes->value(); // Array of: target, title, etc.
$wrapper->field_my_link->url = 'http://mediacurrent.com';
$wrapper->field_my_link->title = 'Do Drupal Right';

// Radio and Checkbox
$v = $wrapper->field_my_radio->value(); // on = 1, off = 0
$info = field_info_field('field_my_radio'); // on = New, off = Used
$allowed_values = $info['settings']['allowed_values'];
$label = $allowed_values[$v]; // on = New, off = Used

// Delete a value, unset a field, taxonomy in selectlist is different
$wrapper->field_foobar = NULL;

// Save
$wrapper->save();

// Select list of taxonomy or entity reference, General Usage
Append index
`$term = $_wrapper->field_taxonomy_a[0]->value();`
If only one value is allowed, it doesn't have index.
Loop
foreach ($_wrapper->field_list_of_values as $item) {
//  do something
}

// Taxonomy in select list
For taxonomy term field which can be multiple values, you still need to specify the index
`$term = $_wrapper->field_taxonomy_a[0]->value();` // this might be wrong..

Multiple Tax Terms 

$a = $wrapper->$field->value();
if (isset($a)) {
  echo $a[0]->name;
  echo $a[0]->tid;
}

Single Tax Term 

$a = $wrapper->$field->value();
if (isset($a)) {
  echo $a->name;
  echo $a->tid;
}

Multiple Files 

$a = $wrapper->$field->value();
if (isset($a)) {
  $r=$a;
  foreach ($r as $k => $v) {
    $r[$k]['url'] = file_create_url($r[$k]['uri']);
    // convert URI to public URL
    // Drupal doesn't provide public 'url'
  }
  echo $r[0]['uri'];
  echo $r[1]['uri'];
}

Term id of the first value of a multiple value field $tid = $_wrapper->field_taxonomy_a[0]->raw(); For taxonomy term field which can only have one value: Term ID: $_wrapper->field_tax_1->raw(); Check empty if (isset($tid))

Term name: $_wrapper->field_tax_1->name->value(); Or in a multi value list $v = $_wrapper->field_tax_1[0]->value(); echo $v->name;

Add a term or entity reference: $_w->field_tax_1[] = 42; $_w->save(); Set as a whole $_w->field_tax_1->set(array(42,43)); $_w->save();

Change a term for a single tax field $_w->field_tax_1->set(42); $_w->save();

A required field can't be unset. Refer to d7:db:delete required field Unset a term for a single tax field unset($_w->field_tax_1) Unset a term for a multi tax field 

foreach($wrapper->field_foobar as $delta => $item)  {
  if ($item->nid->value() == $my_id) {
    unset($wrapper->field_foobar[$delta]);
    $wrapper->save();
    break;
  }
}

Set empty value for tax term field 

$wrapper->field_example_multiple->set();
$wrapper->field_example_multiple = array();
$wrapper->field_example_multiple = NULL;

Loop 

foreach ($_wrapper->field_industry as $taxonomy_wrapper) {
  $industry = $taxonomy_wrapper->tid->value();
}

A list of text 

// All values in a list text as an array
$wrapper->field_list_text->value();

foreach ($wrapper->field_taxonomy_terms->getIterator() as $delta => $term_wrapper) {
  // $term_wrapper may now be accessed as a taxonomy term wrapper.
  $label = $term_wrapper->name->value();
  // list text, delta is 0, 1
  // List text value $term_wrapper->value();
}

If the list of text field only allows one value, then the value is $w->field_list_text->value(); The same as getting a text field

// Entity reference in select list Single value. Tid is $tid = $wrapper->field_list_entity->raw(); Check empty if (isset($tid)) { echo $wrapper->field_list_entity->value()->title} Multiple values.

https://www.drupal.org/documentation/entity-metadata-wrappers http://www.mediacurrent.com/blog/entity-metadata-wrapper

2.23. Entityform

D7 uses entityform module. D8 is called EForm. Load an entityform 

// Load the module file. Optional
// module_load_include('inc', 'entityform', 'entityform-admin');
$_contact_form = entityform_empty_load('entityform_id');
$mode = 'submit'; 
// Default. Other: 'edit'
$form_context = 'page'; 
// Default. Ohter: 'embedded'
$_renderable_form = entityform_form_wrapper($_contact_form, $mode, $form_context );

Try to prefill entityform in hook_form_FORM_ID_alter(). Say the entityform is named dealer_contact The hook is lili_form_dealer_contact_entityform_edit_form_alter

Use modal in CTools, you will need t: 

<a href="/modal/entityform/YOUR_FORM_IDENTIFIER/nojs/0" 
   class="ctools-use-modal ctools-modal-modal-popup-small">Open Entity Form in Popup!</a>

2.24. User

2.24.1. Get user fields

global $user

$account = user_load($user->uid) to load all fields

or use d7:entity_metadata_wrapper to get a couple fields

$user_wrapper = entity_metadata_wrapper('user', $user); drupal_set_message('<pre>'.var_export($user_wrapper->field_node->raw(),1).'</pre>'); // Raw value drupal_set_message('<pre>'.var_export($user_wrapper->field_node->value(),1).'</pre>');

2.24.2. user_access d7:user_access

user_access($string, $account = NULL) $account

  • NULL. current logged in user

2.24.3. user_is_logged_in, user_has_role

user_is_logged_in && user_has_role(3) :: check if current user is admin

anonymous :: 1 authenticated user :: 2 administrator :: 3

2.25. Menu d7:menu

2.25.1. hook_menu: 'page callback'

  • You can print or echo some rendered HTML and then drupal_exit() or exit()
    drupal_exit()
    prevents hook_exit() from running
    (no term)
    Or print and echo some HTML and return NULL;
    (no term)
    No header or footer will be added. The page is pure what you output here.
  • You can return a renderable array. Header and footer will be included
  • You may theme the form by $form['#theme'][] = 'custom_theme_name'; and return $form
  • return theme('custom_theme_name', ['vars'=>...]);
  • In page callback function, you can manipulate some variables used in html.tpl.php and page.tpl.php
    • Such as change the page title drupal_set_title
  • Refer to d7:functions

2.25.2. hook_menu: 'file'

You can local the page callback function in a different file. e.g. 'lili.pages.inc' The path is the current module path

2.25.3. hook_menu: 'type'

MENU_CALLBACK
A hidden internal callback. Best for return plain text in API calls
MENU_LOCAL_ACTION
An action specific to the parent, usually rendered as a link
MENU_LOCAL_TASK
A task specific to the parent item, usually rendered as a tab
MENU_NORMAL_ITEM
Shown in menu and breadcrumbs
MENU_SUGGESTED_ITEM
A normal menu item, hidden until enabled by an administrator

2.25.4. hook_menu: 'access callback', 'access arguments'

  • 'access callback'
    • default is 'user_access' d7:user_access
    • custom callback must return boolean
    • Often used callbacks: 'user_is_logged_in'
    • TRUE is for anyone to access
  • 'access arguments'
    • e.g. ['administer nodes']

2.25.5. Embed a View Page

function lili_menu() {
  $items['path-different-from-view/%'] = array(
    'title' => t('page title');
    'page callback' => 'views_embed_view',
    'page arguments' => array('view_id', 'dispaly_id', 1),
    'type' => MENU_CALLBACK,
    'access callback' => TRUE,
  );
  return $items;
}

2.25.6. Array to CSV

page callback 

drupal_add_http_header('Content-Type', 'text/csv; utf-8');
drupal_add_http_header('Content-Disposition', 'attachment;filename=csvfile.csv');

$fp = fopen('php://output', 'w');
fputcsv($fp, [$year. ' Ranking of Top 100 Carriers']);
$_columns = '#,Company,Web Site,Total,Trucks,Tractors,Trailers,O/OS,Employees';
$_columns = explode(',',$_columns);
fputcsv($fp, $_columns);

foreach ($data as $d) {
  $line   = [ ];
  $line[] = $d->this_rank;
  $line[] = $d->company_name;
  $line[] = $d->web_address;
  $line[] = $d->total_vehicles;
  $line[] = $d->trucks;
  $line[] = $d->tractors;
  $line[] = $d->trailers;
  $line[] = $d->owner_operators;
  $line[] = $d->full_time_employees;

  fputcsv( $fp, $line );
}

fclose($fp);
drupal_exit();

2.25.7. Json

function lili_menu() {
  $items['newsletter/archive/%/%'] = array(
    'title' => "",
    'page callback' => '_lili_archive_by_newsletter_json',
    'page arguments' => array(2,3),
    'access callback' => TRUE,
    'type' => MENU_CALLBACK,
  );

  return $items;
}

function _lili_archive_by_newsletter_json($pub_id,$newsletter_name, $page = 1, $items = 20) {
  drupal_add_http_header('Access-Control-Allow-Origin','*'); // CORS            
  $_result['total'] = '...';
  $_result['items'] = array('...');

  drupal_json_output($_result);
  //return 'hello';
}

2.26. Taxonomy

A node or an entity has a taxonomy term means the node has a category. Categories may have hierarchy.

2.26.1. Get taxonomy terms of a node, ready for display with language

$categories = field_get_items('node', $node, 'field_category');
array(
0 => array(
  'tid' => 17,
  'taxonomy_term' => obj
),
...
)

2.26.2. Get multiple terms by id, taxonomy_term_load_multiple

$tids = array(1,2,3); // Default:array()
$conditions = array(); // Default
taxonomy_term_load_multiple($tids, $conditions);
array(
17 => obj(
  tid => '17',
  ...
)
...
)

2.26.3. Get term by name, taxonomy_get_term_by_name

// taxonomy_get_term_by_name($name, $vocabulary = NULL)

$_limit_to_vocab = 'vocab_name'; // Default NULL.
$term_is_parent = taxonomy_get_term_by_name('video', $_limit_to_vocab);

// load the term's field values
$term_is_parent_id = (!empty($term_is_parent)) ? key($term_is_parent) : 0;
$field_my_field = ($term_is_parent_id) ? field_get_items( 'taxonomy_term', $term_is_parent[$term_is_parent_id], 'field_my_field');

// refer to d7:field_get_items

if (!empty($field_my_field)) {
    foreach ($field_my_field as $k => $v) {
        echo $v['value'];
    }
}

2.26.4. Get all children of a term, taxonomy_get_children

Get a term that is parent 

$_limit_to_vocab = 'vocab_name'; // Default NULL.
$term_is_parent = taxonomy_get_term_by_name('video', $_limit_to_vocab);

// Get the first item in array
$term_is_parent = reset($term_is_parent);

echo $term_is_parent->tid; // parent term id

// Get all children of a term
$children = taxonomy_get_children($term->tid);
$children_tids = array_keys($children);

2.26.5. Get a vocabulary by name and all terms

$_vocab = taxonomy_vocabulary_machine_name_load('vocab_name');

// Get all terms of a vocabulary without extra fields
$_terms = taxonomy_get_tree($_vocab->vid);

// Get term ids only of all terms under a vocabulary
$_q = new EntityFieldQuery();
$_term_ids = $_q 
->entityCondition('entity_type', 'taxonomy_term')
->propertyCondition('vid', $_vocab->vid)
->execute();
foreach ($_term_ids['taxonomy_term'] as $term) {
  $term->id;
}

// Get all fields of all terms under a vocabulary
$_terms_with_all_fields = entity_load('taxonomy_term', FALSE, array('vid' => $_vocab->vid));
/* array(
17 => obj(
 tid => '17',
 ...
),
...
)
*/

// Or

$parent = 0; // Default
$max_depth = NULL; // Default
$load_entities = TRUE; // Default:False
$_terms_with_all_fields = taxonomy_get_tree($_vocab->vid, $parent, $max_depth, $load_entities);
array(
0 => obj(
  tid => '17',
  ...
)
...
)

2.26.6. Get terms of a vocabulary ready for form

d7:form:taxonomy field

2.27. Cache

2.27.1. Setting

Configuration > Peformance (/admin/config/development/performance)

  • Always check Cache pages for anonymous users which is cache
    • External cache system e.g. Varnish may still cache the page even when this is disabled
  • Check Cache blocks which is block_cache for logged-in users.
  • Minimum Cache Lifetime which is cache_lifetime
    • If you're not sure, better leave it as 0
    • It applies to all pages and cache objects
    • The value is It's ok to serve cache objects which are stale for this value amount of time
    • The cache clearing is triggered by events e.g. Drupal Cron. If these events are not triggered, the cache will not be regenerated even if the expiration has passed
    • e.g. If it's set to 5 mins and a new post is created, the new post appears on the homepage at least after 5 mins and Drupal Cron is run
    • cache_set() with no $expire time will take cache_lifetime. Set $expire in cache_set to overide.
    • When cache_lifetime is reached in Redis, cache key and value will be removed from Redis.
  • Start with 15 mins and go up as you need for Expiration of cached pages which is page_cache_maximum_age
    • It has nothing to do with cache objects or Drupal Database
    • It sets an HTTP response header Cache-Control: public, max-age=<...>. Refer to header:cache-control:public max-age

Bandwidth Optimization:

Compress cached pages
If your server (Pantheon) already delivers content in gzip, then don't check it
(no term)
Always check Aggregate and compress CSS files and Aggregate JavaScript files

External cache system may include Varnish and other proxy servers.

2.27.2. Same page request

function lili_cache($field, $set = NULL) {
  $custom_cache = &drupal_static(__FUNCTION__);
  if (!isset($custom_cache)) {
    $custom_cache = array();
  }
  if ($set !== NULL && $field !== NULL ) {
    // $set_field_value = lili_cache('field_name', 123);
    $custom_cache[$field] = $set;
  } 
  elseif ($field !==  NULL && $set == NULL &&
  ( !isset($custom_cache[$field]) || $custom_cache[$field] == NULL ) 
     ) {
    // Set default
    // $set_field_value = lili_cache('field_name');
    switch ($field) {
  case "field_name":
    // complicated
    $custom_cache[$field] = '';
    break;
  case "truck_makes_array":
    $_terms = lili_cache('truck_makes_obj');
    $a = [];
    foreach ($_terms as $k => $v) {
      $a[$v->name] = (tid) $v->tid;
    }
    $custom_cache[$field] = $a;
    break;
  case "truck_makes_obj":
    $vocab = taxonomy_vocabulary_machine_name_load('truck_makes');
    $terms = taxonomy_get_tree($vocab->vid);
    $custom_cache[$field] = (!empty($terms)) ? $terms : [];
    break;
    } 
  }

  if (!isset($custom_cache[$field])) {
    $custom_cache[$field] = NULL;
  }

  return $custom_cache[$field];
}

// Set
lili_cache('current_node',$node);
// Get lili_cache('current_node');

2.27.3. Cache variables in cache table, and clear them

function my_module_function() {
  $my_data = &drupal_static( __FUNCTION__ );
  if ( ! isset( $my_data ) ) {
    if ( $cache = cache_get( 'my_module_data' ) ) {
  $my_data = $cache->data;
    } else {
  // Do your expensive calculations here, and populate $my_data
  // with the correct stuff..
  cache_set( 'my_module_data', $my_data, 'cache' );
  // Set cache with an expiry time
  // cache_set('my_module_data', $my_data, 'cache', time() + 360);
    }
  }
  return $my_data;
}
Clear one variable cache
cache_clear_all('my_module_data', 'cache');
Clear all variables start with my_module
cache_clear_all('my_module', 'cache', TRUE);

2.27.4. Cache block d7:cache block

If there're modules setup node_grants, variable block_cache in Performance will be FALSE and disabled. You will see table cache_block is empty. In Pantheon, cache is in Redis Here's how to Enable caching for specific blocks only. In settings.php 

$conf['block_cache_bypass_node_grants'] = "TRUE";
$conf['block_cache'] = "TRUE";

Set all blocks to DRUPAL_NO_CACHE (_block_get_cache_id($block)) Set the blocks to cache to have any one of

  • DRUPAL_CACHE_GLOBAL
  • DRUPAL_CACHE_PER_PAGE
  • DRUPAL_CACHE_PER_ROLE
  • DRUPAL_CACHE_PER_USER
  • DRUPAL_CACHE_CUSTOM (in this case you need to setup own caching in hook_block_view or hook_block_view_MODULE_DELTA)

But not DRUPAL_NO_CACHE

2.28. File

Read local or remote file to an object of strings 

$result = drupal_http_request($url);
/*
obj(
 code // int 200
 headers
)
*/

$path = 'public://afolder/';
$replace = FILE_EXISTS_RENAME; // Default
// FILE_EXISTS_REPLACE, FILE_EXISTS_ERROR
file_unmanaged_save_data($result->data, $path, $replace);

file_unmanaged_save_data($data, $destination, $replace) 

// $destination is folder path not folder path + filename
$temp_name = 'temporary://auniquefilename'
file_put_contents($temp_name,$data); // return number of bytes or false
file_unmanaged_move($temp_name,$destination, $replace); // rename

Drupal download file with real file extension 

/**
 * Download external file to temporary filesystem
 * with correct file extension based on content-type in http response header
 *
 * @param $url string Full external URL of a file
 *
 * @return string Local file path with correct file extension
 */
function _lili_save_file_with_extension( $url ) {
  $filename  = $url;
  $extension = '';
  $result    = drupal_http_request( $url );

  if ( $result->code == 200 && isset( $result->headers['content-type'] ) ) {
    $mime_type_extension = array(
  'image/jpeg'  => 'jpg',
  'image/jpg'   => 'jpg',
  'image/png'   => 'png',
  'image/x-png' => 'png',
  'image/gif'   => 'gif'
    );

    foreach ( $mime_type_extension as $k => $v ) {
  if ( strcasecmp( $k, $result->headers['content-type'] ) == 0 ) {
    $extension = $v;
    break;
  }
    }

    if ( $extension !== '' ) {
  // Save file with correct file extension
  $filename = 'temporary://import_' . str_replace( ".", "_", microtime( TRUE ) ) . "." . $extension;
  file_put_contents( $filename, $result->data );
    }
  }

  return $filename;
}

2.29. File Permissions d7:file permissions

  • sudo ./fix_drupal_permissions.sh --drupal_path=/var/www/a.ca/public --drupal_user=www-data
#!/bin/bash

# Help menu
print_help() {
cat <<-HELP
This script is used to fix permissions of a Drupal installation
you need to provide the following arguments:

  1) Path to your Drupal installation.
  2) Username of the user that you want to give files/directories ownership.
  3) HTTPD group name (defaults to www-data for Apache).

Usage: (sudo) bash ${0##*/} --drupal_path=PATH --drupal_user=USER --httpd_group=GROUP
Example: (sudo) bash ${0##*/} --drupal_path=/usr/local/apache2/htdocs --drupal_user=john --httpd_group=www-data
HELP
exit 0
}

if [ $(id -u) != 0 ]; then
  printf "**************************************\n"
  printf "* Error: You must run this with sudo or root*\n"
  printf "**************************************\n"
  print_help
  exit 1
fi

drupal_path=${1%/}
drupal_user=${2}
httpd_group="${3:-www-data}"

# Parse Command Line Arguments
while [ "$#" -gt 0 ]; do
  case "$1" in
    --drupal_path=*)
    drupal_path="${1#*=}"
    ;;
    --drupal_user=*)
    drupal_user="${1#*=}"
    ;;
    --httpd_group=*)
    httpd_group="${1#*=}"
    ;;
    --help) print_help;;
    *)
  printf "***********************************************************\n"
  printf "* Error: Invalid argument, run --help for valid arguments. *\n"
  printf "***********************************************************\n"
  exit 1
  esac
  shift
done

if [ -z "${drupal_path}" ] || [ ! -d "${drupal_path}/sites" ] || [ ! -f "${drupal_path}/core/modules/system/system.module" ] && [ ! -f "${drupal_path}/modules/system/system.module" ]; then
  printf "*********************************************\n"
  printf "* Error: Please provide a valid Drupal path. *\n"
  printf "*********************************************\n"
  print_help
  exit 1
fi

if [ -z "${drupal_user}" ] || [[ $(id -un "${drupal_user}" 2> /dev/null) != "${drupal_user}" ]]; then
  printf "*************************************\n"
  printf "* Error: Please provide a valid user. *\n"
  printf "*************************************\n"
  print_help
  exit 1
fi

cd $drupal_path
printf "Changing ownership of all contents of "${drupal_path}":\n user => "${drupal_user}" \t group => "${httpd_group}"\n"
chown -R ${drupal_user}:${httpd_group} .

printf "Changing permissions of all directories inside "${drupal_path}" to "rwxr-x---"...\n"
find . -type d -exec chmod u=rwx,g=rx,o= '{}' \;

printf "Changing permissions of all files inside "${drupal_path}" to "rw-r-----"...\n"
find . -type f -exec chmod u=rw,g=r,o= '{}' \;

printf "Changing permissions of "files" directories in "${drupal_path}/sites" to "rwxrwx---"...\n"
cd sites
find . -type d -name files -exec chmod ug=rwx,o= '{}' \;

printf "Changing permissions of all files inside all "files" directories in "${drupal_path}/sites" to "rw-rw----"...\n"
printf "Changing permissions of all directories inside all "files" directories in "${drupal_path}/sites" to "rwxrwx---"...\n"
for x in ./*/files; do
  find ${x} -type d -exec chmod ug=rwx,o= '{}' \;
  find ${x} -type f -exec chmod ug=rw,o= '{}' \;
done
echo "Done setting proper permissions on files and directories"

2.30. Image Handling (Core)

2.30.1. Setting

Define Image Styles /admin/config/media/image-style Use it in image field display or in code. Image toolkit /admin/config/media/image-toolkit reduces the JPEG quality if Image Styles is applied.

Both Image Styles and Image toolkit don't modify the original images but instead create new images in real time.

2.30.2. Code

An image is a file. Get a image field: 

$w = entity_metadata_wrapper('node', $node);
$images = $w->field_images->value();
if (isset($images)) {
  // If the image field is set to only hold one image, then
  // $images is ['uri'=>'...']
  foreach ($r as $k => $v) {
    $r[$k]['url'] = file_create_url($v['uri']);
    // convert internal URI to public URL. Drupal doens't provide public url by default.
    // This step doesn't create a file, just create a link
    // Get the image style public url
    $r[$k]['thumbnail'] = image_style_url('thumbnail', $v['uri']);
    // It creates an image styled link in db, when requesting the public url, the image will be created
  }
}

2.30.3. Get File URI from db d7:db:file uri

2.31. Basic Ajax

// Create a menu to receive ajax
$items['newsletter/builder_entry/%'] = array(
    'title' => "",
    'page callback' => '_lili_add_session_handler',
    'page arguments' => array(2),
    'access callback' => 'user_access',
    'access arguments' => array('newsletter builder access'),
    'type' => MENU_CALLBACK,
);

function _lili_add_session_handler($command) {
if (!isset($_POST['ids'])) {
    return drupal_not_found();
  }

  if ($command === "manage") {
    // Add to the session variable detailing what items are added
    $_SESSION['session_name'] = array();

    $ids = json_decode($_POST['ids']);
    foreach ($ids as $id) {
  $id = trim($id);
  $_SESSION['session_name']["item_" . $id] = $id;
    }
  }
  else {
    return drupal_not_found();
  }
}

$.ajax({
    type: "POST",
    url: '?q=newsletter/builder_entry/manage',
    data: "ids=" + 'some string',
    success: function(e) {
    // do something
    },
    dataType: "json"
});

2.32. Javascript API

Javascript API Overview

{ 'settings': {}, 'behaviors': {}, 'themes': {}, 'locale': {} }

2.32.1. Settings

Pass PHP variable to javascript

PHP. Use camelCasing 

$my_settings = array(
  'basePath' => $base_path,
  'animationEffect' => variable_get('effect', 'none')
);
drupal_add_js(array('myModule'=>$my_settings), 'setting');

Javascript 

var basePath = Drupal.settings.myModule.basePath;
var effect = Drupal.settings.myModule.animationEffect;

2.32.2. Behaviors

https://www.lullabot.com/articles/understanding-javascript-behaviors-in-drupal https://www.amazeelabs.com/en/blog/drupal-behaviors-quick-how https://www.drupal.org/node/756722

Any property defined in Drupal.behaviors will get called when DOM is ready or when Drupal.attachBehaviors() runs. Drupal.attachBehaviors(); means attach all behaviors. When DOM is ready, Drupal runs: 

// Attach all behaviors.
$(function () {
    Drupal.attachBehaviors(document, Drupal.settings);
});

// Syntax: Drupal.attachBehanviors(context, settings);
// When context is not defined, it's the document object in javascript
// when settings is not defined, Drupal.settings is used
// So Drupal.attachBehaviors(); is the same as Drupal.attachBehaviors(document, Drupal.settings);

Other modules might run multiple Drupal.attachBehaviors(); or Drupal.attachBehaviors(custom_context, custom_settings);

All properties defined in Drupal.behaviors will get called when Drupa.attachBehaviors(…, …); runs! Some common situations:

  • After an admin overlay has been loaded into the page.
  • After AJAX Form API has submitted a form
  • When an AJAX request returns a command that modifies HTML, such as ajax_command_replace()
  • CTools calls it after a modal has been loaded
  • Media calls it after media browser has been loaded
  • Panels calls it after in-place editing has been completed
  • Views calls it after loading a new page that uses AJAX
  • Views Load More calls it after loading the next chunk of items
  • Javascript from custom modules may call Drupal.attahcBehaviors() when they add or change parts of the page.
    • Such as drupal_add_js('jQuery(document).ready(function () { Drupal.attachBehaviors(); });', 'inline');

So you need to attach any event once using $('',context).once() !

This is mymod_path/js/drupal.behaviors.js 

(function ($) {
  Drupal.behaviors.MODULENAME = {
    attach: function (context, settings) {
  console.log('This runs every time Drupal.attachBehaviors(); is called');
  // Run something only once (e.g. set jQuery event)
  $('.nav-flyout', context).once('remove-modals', function () {
    // After this is run, all elements selected will have a new class 'remove-modals-processed'
    $(document).keyup(function (e) {
      if (e.keyCode == 27) {
        $('.nav-flyout', context).removeClass('js-flyout-active');
      }
    });
  });
    }
  }
}(jQuery));

In Drupal 

function hook_preprocess_page(&$vars) {
  // Append a property in Drupal.settings only. And before a behavior is attached and it will be run e.g. when DOM is ready
  drupal_add_js(array('MODULENAME' => array('php_var_1' => variable_get('php_var_1', ''))), 'setting');
  drupal_add_js(drupal_get_path('module', 'MODULENAME') . '/js/drupal.behaviors.js');
}

2.33. Email

Define an email template using hook_mail() 

function lili_mail($key, &$message, $params) {
    // Need to overwrite headers content type if email is html
    $message['headers']['Content-Type'] = 'text/html; charset=UTF-8; format=flowed';

    // $message['from'] is initially set by $from from drupal_mail(). 
    // If not provided, will use sidewide setting
    // $message['to'] and 'language' are set by $to from drupal_mail().

    switch ($key) {
  case 'template_1':
    // Use $params
    $message['subject'] = 'This is subject';
    $message['body'][] = 'Extra';
    $message['headers']['Bcc'] = 'bcc@abc.com';
    break;
    } 
}

// Send email
$module = 'lili'; // module name that the template is defined hook_mail
$to = 'to@abc.com';
$from = 'from@abc.com'; // Default Null
$language = language_default();
$send = TRUE; // Default TRUE 
$params = array(); // Extra params to pass to hook_mail() as $params
// hook_mail_alter can change to false so that it doesn't get sent out
$result = drupal_mail($module, 'template_1', $to, $language, $params, $from, $send);
/*
    $result => array(
    'result' => NULL (cancelled by hook_mail_alter())
    or $system->mail($message) (!$result['result'] is true means fail)
    )
*/

2.34. Life Cycle

  • includes/bootstrap.inc
    • Global variables and functions
    • abstract class DrupalCacheArray
    • class SchemaCache extends DrupalCacheArray
  • drupal_bootstrap( DRUPAL_BOOTSTRAP_FULL );

    • DRUPAL_BOOTSTRAP_CONFIGURATION

      • _drupal_bootstrap_configuration();
      • includes/file.phar.inc
        • drupal_settings_initialize();

      • conf_path() . '/settings.php';

        • If sites/sites.php exists, load it
          • '<port>.<domain>.<path>' => 'directory' e.g.
          • for alias/site 8080.www.drupal.org.mysite.test, confg path is sites/example.com
          • for alias/site a.com, config path is sites/a
          • Try to match alias with $_SERVER['SCRIPT_NAME'] and $_SERVER['HTTP_HOST']
        • If no sites/sites.php or no match, return sites/default
        includes/request-sanitizer.inc
        class DrupalRequestSanitizer
        (no term)
        DrupalRequestSanitizer::sanitize();

    • DRUPAL_BOOTSTRAP_PAGE_CACHE

      _drupal_bootstrap_page_cache();
      Attempts to serve a page from the cache
      includes/cache.inc
      functions, class DrupalCacheInterface, DrupalDatabaseCache implements DrupalCacheInterface
      $conf['cache_backends']
      include each cache_backend
      drupal_bootstrap(DRUPAL_BOOTSTRAP_VARIABLES, FALSE);
      bootstrap variables
      (no term)
      If there is no session cookie and cache is enabled (or forced), try to serve a cached page

    • DRUPAL_BOOTSTRAP_DATABASE

      _drupal_bootstrap_database();
      Initializes the database system and registers autoload functions
      • includes/database/database.inc

    • DRUPAL_BOOTSTRAP_VARIABLES

      _drupal_bootstrap_variables();
      Loads system variables and all enabled bootstrap modules
      variable_get('lock_inc', 'includes/lock.inc')
      functions
      includes/module.inc
      functions
      module_load_all(TRUE);
      Loads all the modules that have been enabled in the system table
      foreach (module_list(TRUE, $bootstrap) as $module) { drupal_load('module', $module); }
      load bootstrap modules only (for returning cache content)
      (no term)
      Sanitize $_GET['destination'] and $_REQUEST['destination']
    • DRUPAL_BOOTSTRAP_SESSION
      variable_get('session_inc', 'includes/session.inc')
      functions
      drupal_session_initialize()
      Initializes the session handler, starting a session if needed

    • DRUPAL_BOOTSTRAP_PAGE_HEADER

      _drupal_bootstrap_page_header();
      Invokes hook_boot(), initializes locking system, and sends HTTP headers
      bootstrap_invoke_all('boot');
      Invokes a bootstrap hook in all bootstrap modules that implement it
    • DRUPAL_BOOTSTRAP_LANGUAGE
      drupal_language_initialize();
      Initializes all the defined language types

    • DRUPAL_BOOTSTRAP_FULL

      includes/common.inc
      Global variables and functions
      (no term)
      _drupal_bootstrap_full();
      variable_get('path_inc', 'includes/path.inc');
      functions
      includes/theme.inc
      global variables, class ThemeRegistry extends DrupalCacheArray and functions
      includes/pager.inc
      class PagerDefault extends SelectQueryExtender and functions
      variable_get('menu_inc', 'includes/menu.inc')
      global variables and functions
      includes/tablesort.inc
      class TableSort extends SelectQueryExtender and functions
      includes/file.inc
      global variables and functions
      includes/stream_wrappers.inc
      global variables and classes
      includes/unicode.inc
      global variables and functions
      includes/image.inc
      functions
      includes/form.inc
      functions
      includes/mail.inc
      global variable, class MailSystemInterface and functions
      includes/actions.inc
      functions
      includes/ajax.inc
      functions
      includes/token.inc
      functions
      includes/errors.inc
      functions
      module_load_all();
      load system_list('module_enabled') modules
      drupal_path_initialize();
      Initialize the $_GET['q'] variable to the proper normal path
      menu_set_custom_theme();
      drupal_theme_initialize();
      module_invoke_all('init');
      Invokes a hook_init in all enabled modules
  • Execute the page callback associated with the current path
    hook_menu_site_status
    drupal_alter('menu_site_status', ...); Control site status before menu dispatching
    hook_menu_get_item
    drupal_alter('menu_get_item', $router_item, $path, $original_map);
    (no term)
    $router_item['include_file']
    (no term)
    call_user_func_array($router_item['page_callback'], $router_item['page_arguments'])
    (no term)

    drupal_deliver_page($page_callback_result, $default_delivery_callback);

    hook_page_delivery_callback_alter
    drupal_alter('page_delivery_callback', $delivery_callback);
    drupal_deliver_html_page
    default deliver callback
    • drupal_render_page($page_callback_result)
      • hook_page_build()
      • drupal_alter('page', $page);

      • return drupal_render( $page );

        • theme()
        • template_preprocess($default_template_variables, $hook);
          template_preprocess_HOOK(&$variables)
          Should be implemented by the module that registers the theme hook, to setup default variables
          MODULE_preprocess()
          hook_preprocess()
          MODULE_preprocess_HOOK()
          hook_preprocess_HOOK() is invoked on all implementing modules, so that modules that didn't define the theme hook can alter the variables
          (no term)
          ENGINE_engine_preprocess()
          (no term)
          ENGINE_engine_preprocess_HOOK()
          THEME_preprocess()
          Allows the theme to set necessary variables for all theme hooks with template implementation
          THEME_preprocess_HOOK()
          Allows the theme to set necessary variables specific to the particular theme hook template_process()
          template_process_HOOK()
          Should be implemented by the module that registers the theme hook, if it needs to perform additional variable processing after all preprocess functions have finished
          MODULE_process()
          It's invoked on all implementing modules
          MODULE_process_HOOK()
          It's invoked on all implementing modules, so that modules that didn't define the theme hook can alter the variable
          (no term)
          ENGINE_engine_process()
          (no term)
          ENGINE_engine_process_HOOK
          THEME_process()
          Allows the theme to process the vars for all theme hooks with template implementations
          THEME_process_HOOK()
          Allows the theme to process the vars specific to the theme hook
    • drupal_page_footer();
      hook_exit()
      module_invoke_all('exit');
      (no term)
      serve and write cache
      Run cron
      system_run_automated_cron();

2.35. Hook System d7:hook system

  • Name
    Core
    page, node, taxonomy_term
  • Invoke
    drupal_alter($type, &$data, &$context = NULL, &$context2 = NULL, &$context3 = NULL)

    Pass alterable variables to specific hook_TYPE_alter() implementations

    • Max 2 alterable arguments is supported ($context3 is not supported anymore)
    • In case more arguments need to be passed and alterable, use $context2 and leave $context3 undefined
    $context = array(
  'alterable' => &$alterable,
  'unalterable' => $unalterable,
  'foo' => 'bar',
);
drupal_alter('mymodule_data', $alterable1, $alterable2, $context);

// note that if any object are passed in `$context`, it will be passed by reference in PHP5
// If it's required that there's no implementation alters a passed object in $context, clone the object and pass
$context = array(
  'unalterable_object' => clone $object,
);
drupal_alter('mymodule_data', $data, $context);
    $type
    string or arry e.g. 'form' 'links' 'node_content'. If it's array, hook_TYPE_alter() is invoked for each value in array, ordered first by module, and then for each module, in the order of values in $type. e.g. Form API ['form', 'form_'.$fomr_id] to execute hook_form_alter() and then hook_form_FORM_ID_alter()
    $data
    to be altered
    module_invoke_all($hook)
    Invoke a hook in all enabled modules. Variables are passed by value not by reference
    $hook
    hook name. call_user_func_array($module . '_' . $hook, $args)
    $args
    Arguments to pass to the hook
    module_invoke($module, $hook)
    invokes a hook in a particular module. All arguments are passed by value
  • Hook order
    Module weight

    default 0. The lower the higher priority. Can be negative numbers

    Check all module weights
    print implode(PHP_EOL, module_list());
    SQL update
    UPDATE system SET weight=999 WHERE name = 'mymodule'
    Per hook
    hook_module_implements_alter(&$implementations, $hook). Hook is invoked during module_implements()
    $implementations
    Array keyed by the module's name. The value of each item corresponds to a $group, which is usually FALSE, unless the implementation is in a file names $module.$group.inc
    $hook

    The name of the module hook being implemented 

    unction hook_module_implements_alter(&$implementations, $hook) {
    if ($hook == 'rdf_mapping') {

        // Move my_module_rdf_mapping() to the end of the list. module_implements()
        // iterates through $implementations with a foreach loop which PHP iterates
        // in the order that the items were added, so to move an item to the end of
        // the array, we remove it and then add it.
        $group = $implementations['my_module'];
        unset($implementations['my_module']);
        $implementations['my_module'] = $group;
    }
}

// always execute first for hook_form_alter and form_FORM_ID_alter
function my_module_module_implements_alter(&$implementations, $hook) {
    if (($hook == 'form_FORM_ID_alter' || $hook == 'form_alter') && isset($implementations['module_name']) {
        $module = 'my_module';
        $group = array($module => $implementations[$module]);
        unset($implementations[$module]);
        $implementations = $group + $implementations;

    }


    if ($hook != 'hook_form_alter') {
        return;
    }
}
    module .install file
    d7:hook_install
    Use contributed modules
    module_weight and Utility
    (no term)
    Module name alphabetical order

2.36. OOP

  • D7 refers to oop_examples module
  • One quick way to include a class is to put the class inside any file e.g. $moddir/lib/reader.inc and include when it's needed using module_load_include('inc', 'modename', 'lib/reader');

3. WordPress

3.1. Core Update

Fine tune 

add_filter( 'allow_dev_auto_core_updates', '__return_true' );           // Enable development updates 
add_filter( 'allow_minor_auto_core_updates', '__return_true' );         // Enable minor updates
add_filter( 'allow_major_auto_core_updates', '__return_true' );         // Enable major updates

3.1.1. Manual Update Core

3.1.2. Release

See if wp-config.php .htaccess files are modified 3.8.6 > 4.5.10 > (db) 4.9.1 > 4.9.4

3.1.3. TS: update-core.php blank

Add this to disable WP to check FTP
define('FS_METHOD', 'direct');
(no term)
Refer to wp:ftp https://stackoverflow.com/questions/28598547/wordpress-core-update-fail-4-1-to-4-1-1

3.2. Migrate database

At your previous host, access phpMyAdmin. Load your WordPress database by clicking the database name in the left hand menu pane. Click the "export" tab at the top right. Under "export, highlight all the tables and select "SQL". These are the default settings. Under "options" make sure that "add DROP TABLE/ VIEW/ PROCEDURE/ FUNCTION" is selected. Ensure that "Save as File" towards the bottom of the page is selected. Click Go

For import, just hit Import tab and import. Watch out for the timeout limit.

Change website url Open wp_options table, change siteurl to http://yourwebsite.com

Do it on code level 

define('WP_HOME','http://'.$_SERVER[HTTP_HOST].'/');
define('WP_SITEURL','http://'.$_SERVER[HTTP_HOST].'/');

Go to wp-admin > Settings > Permalink and save to update the permalink

3.3. Redirect a path a Wordpress website wp:redirect path to wp

abc.com/enterprise to another Wordpress website abc.com DB is abc_enterprise and abc

abc.com is inside a docker container using apache2

abc.com uses Nginx and initial setup is 

server {
   listen 80;
   server_name abc.com www.abc.com;

   location / {
  proxy_set_header   X-Real-IP   $remote_addr;
  proxy_set_header   Host        $http_host;
  proxy_pass         http://127.0.0.1:9977;
   }

}

3.3.1. Method 1: Apache

Directly create or copy a sub directory under abc.com root /var/www/html/enterprise

Create a new database called abc_enterprise and import the db /var/www/html/enterprise/wp-config.php

If the enterprise site is a complete new wp installation, you don't need to do this 

define( 'WP_HOME', '/enterprise' );
define( 'WP_SITEURL', '/enterprise' );

// for localhost
// define('WP_HOME','http://'.$_SERVER[HTTP_HOST].'/enterprise');
// define('WP_SITEURL','http://'.$_SERVER[HTTP_HOST].'/enterprise');

// Change the db setting
define('DB_NAME', 'abc_enterprise');

var/www/html/enterprise.htaccess 

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /enterprise/
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /enterprise/index.php [L]
</IfModule>

# END WordPress

Then go to wp-admin and save the permalink

abc.com/enterprise/abc has $_SERVER['REQUEST_URI'] appear as /abc on /var/www/html/enterprise/index.php

  • Redirect abc.com/xyz to abc.com/enterprise/ijk use var/www/html.htaccess
RedirectMatch 302 ^/psup/?$ /enterprise/ijk/
  • Rewrite abc.com/ijk to abc.com/enterprise/real-page so that URL remains abc.com/ijk in address bar

use var/www/html.htaccess 

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^ijk/?(.*)$ /enterprise/ijk [L]
</IfModule>

Then in /var/www/html/enterprise/wp-content/themes/your-theme/functions.php

NOTE: $_SERVER['REQUEST_URI'] is /ijk in ./enterprise/index.php 

function cs_rewrite_request($query){

  $request_uri = urldecode($_SERVER['REQUEST_URI']);
  //var_dump($request_uri);
  /* for pages */
  if( 0 === strpos($request_uri, '/ijk') ){
    $query['pagename'] = urlencode('real-page');
    unset($query['name']);
  }

  return $query;
}
add_filter( 'request', 'cs_rewrite_request', 9999, 1 );

If you want abc.com/enterprise/real-page to redirect to abc.com/ijk which shows the real-page var/www/html.htaccess 

RedirectMatch 302 ^/enterprise/real-page/?$ /ijk/

Fix image urls 

UPDATE wp_posts SET post_content=(REPLACE (post_content, '<old url>','<new url>'));
UPDATE wp_posts SET post_content=(REPLACE (post_content, 'abc-ent-dev.com','abc.com/enterprise'));

3.3.2. Method 2: (Not working..) Nginx proxy to another

Refer to nginx:redirect path to proxy

abc.com Nginx becomes 

server {
   listen 80;
   server_name alectraconservation.com www.alectraconservation.com;

   location ^~ /enterprise/ {
     proxy_set_header   X-Real-IP   $remote_addr;
     proxy_set_header   Host        $http_host;
     proxy_pass http://127.0.0.1:8787/;
   }

    location / {
  proxy_set_header   X-Real-IP   $remote_addr;
  proxy_set_header   Host        $http_host;
  proxy_pass         http://127.0.0.1:9977;
   }

}

Refer to http://www.ur-ban.com/2015/07/27/nginx-proxy_pass-wordpress-in-a-sub-directory/ Change WP_HOME and WP_SITEURL 

$_SERVER['REQUEST_URI'] = str_replace("/wp-admin/", "/enterprise/wp-admin/",  $_SERVER['REQUEST_URI']);
define( 'WP_SITEURL', '/enterprise' );
define( 'WP_HOME', '/enterprise' );

3.4. Load WP Core in sub directory under same domain

sub directory has a customized PHP application and it needs to use WP Core to determine if a user is logged in

https://www.webhostinghero.com/wordpress-authentication-integration-with-php/ http://dovy.io/wordpress/authenticating-outside-of-wordpress-on-diff-domain/

subfolder/index.php 

define( 'WP_USE_THEMES', false ); // NOT load the theme at all or any of that content.
define( 'COOKIE_DOMAIN', false ); // NOT verify the cookie domain. We just want to take the cookie as it is.
define( 'DISABLE_WP_CRON', true ); // Optional

// load less core files. Good for wp REST API request. e.g. global $wp, $wp_query, $wp_the_query are set to NULL
// define( 'SHORTINIT', TRUE );

// if(!session_id()) session_start(); // I find this not necessary

require_once(__DIR__ . "/../wp-load.php");
// You will find some errors, fix errors in your custom themes and plugins
// clear cache and try again

// see wp:user

3.5. Global Variables

https://codex.wordpress.org/Global_Variables

3.5.1. $wp_version

global $wp_version;
if ( version_compare( $GLOBALS['wp_version'], '4.7-alpha', '<' ) ) {
    require get_template_directory() . '/inc/back-compat.php';
    return;
}

3.5.2. $_GET, $_POST, $_COOKIE, $_SERVER

WP first detects if Magic Quotes, if so, remove them using stripslashes_deep (wp function) and then add slashes using add_magic_quotes which uses php:addslashes function. In order to get values, use stripslashes_deep again 

$value = stripslashes_deep($_POST['name']);
// or simply
$myPost = stripslashes_deep($_POST);

// in custom PHP app, detect if WP already defines the function and later remove the slashes
 $myPost = $_POST;
if (function_exists('stripslashes_deep')) {
  // This means WP has already added slashes to $_POST. Remove slashes
  $myPost = stripslashes_deep($myPost);
}

3.5.3. $wp_query $GLOBALS['wp_query'] wp:global:wp_query

Refer to WP_Query 

$p = $GLOBALS['wp_query'];
// first post $p->posts[0]

// total number of posts found by query
echo $p->found_posts;

// number of posts being displayed
echo $p->post_count;

3.5.4. $wp_rewrite wp:global:rewrite

3.5.5. $post wp:global:post

https://codex.wordpress.org/Class_Reference/WP_Post 

global $post;
$post_slug = $post->post_name;

3.5.6. $wpdb wp:global:wpdb

3.5.6.1. Use $wpdb to create raw query

https://code.tutsplus.com/tutorials/writing-custom-queries-in-wordpress--wp-25510 

global $wpdb;
$query = "
    SELECT *
    FROM wp_terms wt
    INNER JOIN wp_term_taxonomy wtt ON wt.term_id = wtt.term_id
    WHERE wtt.taxonomy = 'post_tag' AND wtt.count = 0";

$wpdb->query($query); // int number of rows affected/selected and false when there is an error

// array.
$wpdb->get_results($query, ARRAY_A);

// get one variable. Useful when query result has only one column
$wpdb->get_var($query);

// e.g.
$user_count = $wpdb->get_var( "SELECT COUNT(*) FROM $wpdb->users" );
echo "<p>User count is {$user_count}</p>";

// return one row. result_type can be OBJECT, ARRAY_A or ARRAY_N (object, associative array or numbered array). Offset is an integer with a default of 0
$wpdb->get_row($query, ARRAY_A, 3);

// get a column. Output will be a dimensional array. Empty array if no result. The second parameter is the column offset
$wpdb->get_col($query, 3);

// to prevent SQL injection, use prepare
$wpdb->query( $wpdb->prepare(
    "INSERT INTO test_table (post_id, animal, food) VALUES ( %d, %s, %s )",
    array(
    10,
    'monkey',
    'apple'
    )
));

$wpdb->show_errors();
$wpdb->hide_errors();

$wpdb->flush();

$wpdb->insert(
    'foods',
    array(
    'fruit' => 'apple',
    'year' => 2012
    ),
    array(
    '%s',
    '%d'
    )
);

$wpdb->update(
    'foods',
    array(
    'fruit' => 'apple',  // string
    'year' =>  'value2'  // integer (number)
    ),
    array( 'ID' => 1 ),
    array(
    '%s',   // value1
    '%d'    // value2
    ),
    array( '%d' )
);

// column info
$wpdb->get_col_info('type', offset);
// Type: the information you want to retrieve, some examples are here
// name – column name (this is the default)
// table – name of the table the column belongs to
// max_length – maximum length of the column
// not_null – 1 if the column cannot be NULL
// more can be found in the WordPress Codex WPDB reference
// Offset: specify the column from which to retrieve information (0 is the first column)

// reference WordPress tables
$wpdb->posts;
$wpdb->postmeta;
$wpdb->comments;
$wpdb->commentmeta;
$wpdb->terms;
$wpdb->term_taxonomy;
$wpdb->term_relationships;
$wpdb->users;
$wpdb->usermeta;
$wpdb->links;
$wpdb->options;

// e.g.
$user_count = $wpdb->get_var( "SELECT COUNT(*) FROM $wpdb->users" );

3.5.7. $current_user wp:global:current_user

3.5.8. $wp_filter, $wp_current_filter wp:global:wp_filter wp:global:wp_current_filter

Use current_action() or current_filter() to return the current filter

3.5.9. $template wp:global:template

Which php file is used for template 

global $template;
print_r( $template );

3.5.10. Image Sizes $_wp_additional_image_sizes

Refer to wp:debug:image sizes

3.5.11. Post statuses $wp_post_statuses wp:global:wp_post_statuses

3.5.12. $wp_customize wp:global:wp_customize

3.6. wp-config.php

https://codex.wordpress.org/Editing_wp-config.php

3.6.1. wp:wp-config:DISALLOW_FILE_EDIT

define( 'DISALLOW_FILE_EDIT', true );

3.7. Options, Settings wp:options

/wp-admin/options.php
show all options. Serialized data is not shown
(no term)
Option Reference
stylesheet
The slug of the currently activated stylesheet (style.css). Which is the active theme's dir name. Also called theme slug wp:options:stylesheet
theme_mods_{$theme_slug}
wp:options:theme_modstheme_slug
sticky_posts
array of post ID's (only post type post)
permalink_structure
e.g. /%postname%/
rewrite_rules
serialized wp:options:rewrite_rules
Custom post type
[ "articles/(.+?)(?:/([0-9]+))?/?$" => "index.php?articles=$matches[1]&page=$matches[2]" ]
Custom taxonomy
[ "neckline/([^/]+)/?$" => "index.php?gallery_neckline=$matches[1]" ]
noindex, nofollow
Settings > Reading > toggle Search Engine Visibility - Discourage search engines from indexing this site

3.8. Email PHPMailer

Add this after require_once(ABSPATH . 'wp-settings.php'); in wp-config.php This way you don't need to have linux:sendmail installed on Linux

Doc 

// configure phpmailer
add_action( 'phpmailer_init', 'mail_relay' );
function mail_relay( $phpmailer ) {
    $phpmailer->isSMTP();
    $phpmailer->Host = 'smtp.gmail.com';
    $phpmailer->SMTPAutoTLS = true;
    $phpmailer->SMTPAuth = true; 
    $phpmailer->Port = 465;
    $phpmailer->Username = '<admin email>';
    $phpmailer->Password = '<app password>';

    // Additional settings
    $phpmailer->SMTPSecure = "ssl"; 
    $phpmailer->From = "<admin email>";
    $phpmailer->FromName = "<your name>";
}
// configure phpmailer.

Without bootstrap wp, assuming test.php file is at root 

error_reporting(E_ALL);
ini_set('display_errors', 1);

echo 'hello';

// PHPMailer Gmail Example
// https://github.com/PHPMailer/PHPMailer/blob/master/examples/gmail.phps

$recipient_email = 'a@b.com';
$recipient_name = 'First Last';
$sender_name = 'Sender Name';
$subject = 'Testing';
$html = ('
    <html>
    <head>
  <title>This is a test</title>
    </head>
    <body>
  <h1>Testing headline</h1> 
  <p>Testing body content</p>
    </body>
    </html>
  ');
$text = 'pure text testing';

// php's native mail
// mail($recipient_email, $subject, $text);

require("wp-includes/class-phpmailer.php");
$mail = new PHPMailer();

$mail->isSMTP(); // Set mailer to use SMTP
//Enable SMTP debugging
// 0 = off (for production use)
// 1 = client messages
// 2 = client and server messages
$mail->SMTPDebug = 2;

$mail->Host = 'smtp.gmail.com';
// use
// $mail->Host = gethostbyname('smtp.gmail.com');
// if your network does not support SMTP over IPv6

$mail->SMTPAuth = true; // set to false if Username, Password, SMTPSecure are empty
$mail->Username = 'username@gmail.com';
$mail->Password = 'password';
$mail->SMTPSecure = 'tls'; // Enable TLS encryption, `ssl` also accepted

$mail->Port = 587; // 587 for tls

$mail->CharSet = 'UTF-8';

$mail->SetFrom('donotreply@b.com', $sender_name);

$mail->AddAddress($recipient_email, $recipient_name);

$mail->Subject = $subject;

//$mail->addReplyTo('replyto@example.com', 'First Last');

//Replace the plain text body with one created manually
// $mail->AltBody = 'This is a plain-text message body';

//Attach an image file
//$mail->addAttachment('images/phpmailer_mini.png');

$mail->MsgHTML($html);

if (!$mail->Send()) {
  echo "Mailer Error: " . $mail->ErrorInfo;
}
else {
  echo "successful!";
}

3.9. PHP redirect wp:php:redirect

Redirect, same can be achieved using .htaccess 

function rudr_url_redirects() {
    /* in this array: old URLs=>new URLs  */
    $redirect_rules = array(
        array('old'=>'/category/uncategorized/','new'=>'/category/Uncategorized/'), // category
        array('old'=>'/contacts/','new'=>'/Contacts/'), // page
        array('old'=>'/hello-world/','new'=>'/hello-planet/'), // post
        array('old'=>'/tag/wordpress/','new'=>'/tag/WordPress/') // post tag
    );
    foreach( $redirect_rules as $rule ) :
        // if URL of request matches with the one from the array, then redirect
        if( urldecode($_SERVER['REQUEST_URI']) == $rule['old'] ) :
            wp_redirect( site_url( $rule['new'] ), 301 );
            exit();
        endif;
    endforeach;
}
add_action('template_redirect', 'rudr_url_redirects');

Redirect by changing request based on the $_SERVER['REQUEST_URI'] 

function rudr_rewrite_request($query){

    $request_uri = urldecode($_SERVER['REQUEST_URI']);

    /* for categories */
    if ( $request_uri == '/category/Uncategorized/' )
        $query['category_name'] = 'uncategorized';

    /* for pages */
    if ( $request_uri == '/Contacts/' ){
        $query['pagename'] = urlencode('contacts');
        unset($query['name']);
    }

    /* for posts */
    if ( $request_uri == '/hello-planet/' )
        $query['name'] = 'hello-world';

    /* for tags */
    if( $request_uri == '/tag/WordPress/' )
        $query['tag'] = 'wordpress';

    return $query;
}
add_filter( 'request', 'rudr_rewrite_request', 9999, 1 );

Change permalink For posts and pages 

function rudr_post_permalink( $url, $post ){
    if( !is_object( $post ) )
        $post = get_post( $post_id );

    $replace = $post->post_name;

    /* We should use a post ID to make a replacement. It is required if you use urf-8 characters in your URLs */

    if( $post->ID == 1 ) 
        $replace = 'hello-planet';
    if( $post->ID == 12 ) 
        $replace = 'Contacts';

    $url = str_replace($post->post_name, $replace, $url );
    return $url;
}

add_filter( 'post_link', 'rudr_post_permalink', 'edit_files', 2 );
add_filter( 'page_link', 'rudr_post_permalink', 'edit_files', 2 );
add_filter( 'post_type_link', 'rudr_post_permalink', 'edit_files', 2 );

For categories and tags 

function rudr_term_permalink( $url, $term, $taxonomy ){
    $replace = $term->slug;

    /* by ID as well */
    if( $term->term_id == 5 ) 
        $replace = 'Uncategorized';
    if( $term->term_id == 55 ) 
        $replace = 'WordPress';

    $url = str_replace($term->slug, $replace, $url );

    return $url;
}

add_filter( 'term_link', 'rudr_term_permalink', 10, 3 );

3.10. WP-CLI wp-cli

3.10.1. WP CLI Install, Upgrade

# Assuming you are at ~ folder
curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar

# Check phar
php wp-cli.phar --info

chmod +x wp-cli.phar

# GoDaddy Shared Hosting doesn't allow sudo. Ignore this for GoDaddy
# sudo mv wp-cli.phar /usr/local/bin/wp
alias wp='~/wp-cli.phar'
echo "alias wp='~/wp-cli.phar'" >> .bashrc
source .bashrc

Set Global parameters in wp-cli.yml under WordPress root folder http://wp-cli.org/config/

Use – in front for wp command 

# Very important. To avoid error: $_SERVER['HTTP_HOST']
url: http://www.yourwebsite.com

Use --skip-plugins --skip-themes to avoid errors while loading plugins/themes that cause this kind of error Fatal error: Uncaught Error: Call to a member function xxx on null

Check wp-cli installation wp --info, wp plugin list --debug or wp plugin list --skip-plugins --debug

If you see this error, time to upgrade WP CLI Fatal error: Class 'WP_Post_Type' not found in /Applications/MAMP/htdocs/wp-includes/post.php on line 1031

sudo wp cli update

3.10.2. Global Parameters

  • –debug
  • –quiet
  • –user=<id|login|email>
  • Load PHP file, can be used multiple times

3.10.3. Command documentation

https://github.com/wp-cli and go to db-command/src/DB_Command.php for command db.

3.10.4. user

# List users
terminus wp 'user list' --site=<site> --env=<env>
terminus wp mysite.live -- user list
lando terminus wp mysite.live user list

wp user list

# Add a user, a random password is assigned
wp user create lili li@xxx.ca --role=administrator

# add a user, prompt for a password
wp user create lili li@xxx.ca --role=administrator

# Update password
wp user update lili --user_pass='lettersandnumbers'

# Prompt for password with special characters
wp user update lili --prompt=user_pass

# reset password (change to a random one and no email will be sent)
wp user reset-password adminname editorname

# Give role
wp user add-role lili administrator

# remove role
wp user remove-role lili subscriber

# in case of `--` cannot be used
# a random password is assigned
wp user create lili li@a.ca
wp user add-role lili administrator
# then reset password from front end

3.10.5. core

wp core download
wp core version
wp core update

3.10.6. plugin

Wordpress Official Plugin Directory 

# List installed plugins
wp plugin list

# install without activate
wp plugin install fly-dynamic-image-resizer

# install and activate
wp plugin install custom-post-type-ui --activate

wp plugin status
wp plugin update akismet
wp plugin activate plugin_name
wp plugin deactivate plugin_name
wp plugin uninstall Plugin_name

lando terminus wp mysite.dev plugin install post-expirator
lando terminus wp mysite.dev plugin activate post-expirator

3.10.7. package

All packages 

# List installed packages
wp package list

# Install a package
wp package install aaemnnosttv/wp-cli-login-command

3.10.8. theme

wp theme status

3.10.9. login

GitHub

user
can be a User ID, username/login or email address.
wp package install aaemnnosttv/wp-cli-login-command

# find a user
wp user list

# create a temp login url for a user
wp login create <user> [options]

# Will ask you to install a plugin which contain only one .php file
wp login install --activate --allow-root

# run create again

# Make all exisiting magic links to fail
wp login invalidate

# Deactivate the plugin
wp login toggle off --allow-root

Options

–expires=<seconds>
default 900 (15 minutes)

3.10.10. media

# regenerate by attachment IDs
wp media regenerate 123 124

# regenerate all thumbnails
wp media regenerate --yes

# regenerate with IDs between
seq 1000 2000 | xargs wp media regenerate

# regenerate image size "large" for all images
wp media regenerate --image_size=large

3.10.11. post wp-cli:post

Documentaion
https://github.com/wp-cli/entity-command
Code
https://github.com/wp-cli/entity-command/blob/master/src/Post_Command.php
3.10.11.1. post list [--<field>=<value>]... [--field=<field>] [--fields=<fields>...]... [--format=<format>]
--post_type and other args in WP_Query can be passed
https://developer.wordpress.org/reference/classes/wp_query/
wp post list --post_type=post --fields=post_title,post_name,post_date,post_status --posts_per_page=5
wp post list --post_type=post --format=count

# list attachment
wp post list --post_type='attachment'

# list posts by post id's
wp post list --post__in=1,3

# list posts by author
wp post list --author__in=1,3

# delete all posts of a post type
wp post delete $(wp post list --post_type='page' --format=ids)

# --force to skip trash and delte permanently

# delete all posts in trash
wp post delete $(wp post list --post_status=trash --format=ids)

# delete all attachments by an author
wp post delete $(wp post list --post_type=attachment --author__in=1,3 --format=ids)

# delete revisions
wp post delete $(wp post list --post_type='revision' --format=ids)

# meta query
wp post list --post_type=post --meta_key=mycustomfield '--meta_compare=NOT EXISTS'
wp post list --post_type=post --meta_key=wpcf-abc --meta_value='abc xyz'
wp post list --post_type=post --meta_key=wpcf-abc --meta_value='abc xyz' '--meta_compare=!='
3.10.11.2. post delete <id>... [--force] [--defer-term-counting]
3.10.11.3. post update <id>... [--field=value]...
Warning!!! Must define --post_name for updating existing posts
Otherwise post_name is set to empty
  • Better to use wp-admin Bulk Edit to trigger events or use raw query to update existing posts
(no term)
It calls wp:f:wp_insert_post
3.10.11.4. post meta wp-cli:post meta
  • Have to find out and set the private field value _custom_order which refers to the wp:plugin:acf field setting in table wp_posts with post_name=field_RANDOM
# wp post meta update <id> <key> <value>
wp post meta update 4045 custom_order 0 _custom_order field_59038b5c9457b

for id in $(wp post list --category_name=arizona-business --fields=ID --format=ids); do wp post meta update $id custom_order 0; done
for id in $(wp post list --category_name=arizona-business --fields=ID --format=ids); do wp post meta update $id _custom_order field_59038b5c9457b; done

# sometimes `wp post list` returns PHP error.. copy those ids and run
for id in 1234 1232 1231; do wp post meta update $id custom_order 0; done

for id in 1234 1232 1231; do lando wp post meta update $id custom_order 0; done


lando ssh --user=root
# get a post with _custom_order field reference
terminus wp mysite.li-dev -- post meta get 12345 _custom_order
# get a list of posts to update custom fields
terminus wp mysite.li-dev -- post list --post_type=speaker --fields=ID --format=ids --allow-root

# fast
terminus wp mysite.li-dev -- post meta update 123 124 custom_order 0
terminus wp mysite.li-dev -- post meta update 123 124 _custom_order field_CHANGERANDOM

# very slow
for id in 1234 1232 1231; do terminus wp mysite.li-dev -- post meta update $id custom_order 0; done
for id in 1234 1232 1231; do terminus wp mysite.li-dev -- post meta update $id _custom_order field_CHANGERANDOM; done

3.10.12. db check

3.10.13. db search

https://developer.wordpress.org/cli/commands/db/search/ 

wp db search 'your search string'
# regex
wp db search 'https?://' --regex --stats

3.10.14. db export

https://developer.wordpress.org/cli/commands/db/export/ 

# Export database with drop query included
$ wp db export --add-drop-table
Success: Exported to 'wordpress_dbase-db72bb5.sql'.

wp db export --add-drop-table /var/www/html/devops/db/dump.sql

3.10.15. db size

wp db size --tables --size_format=mb

3.10.16. db query

wp db query 'SELECT id FROM wp_posts WHERE post_content LIKE "%trans.gif%" AND post_type NOT IN ('revision')'

# In PowerShell, you need to escape "

wp db query 'SELECT id FROM wp_posts WHERE post_content LIKE \"%trans.gif%\" AND post_type NOT IN ('revision')'

3.10.17. search-replace wp-cli:search-replace

Search/replace intelligently handles PHP serialized data, and does not change primary key values. 

# Search and replace but skip one column
wp search-replace 'http://example.dev' 'http://example.com' --skip-columns=guid

# dry run; only for specific tables
wp search-replace 'foo' 'bar' wp_posts wp_postmeta wp_terms --dry-run

# only for specific columns
wp search-replace 'foo' 'bar' wp_posts --include-columns=post_title,post_content

# Run case-insensitive regex search/replace operation (slow)
wp search-replace '\[foo id="([0-9]+)"' '[bar id="\1"' --regex --regex-flags='i'

# Turn your production multisite database into a local dev database
wp search-replace --url=example.com example.com example.dev 'wp_*options' wp_blogs

# Search/replace to a SQL file without transforming the database
# Current database is not changed. The change result is output to a file
wp search-replace foo bar --export=database.sql

# Pantheon Terminus
# For deletion, don't use '', use ' ' with space
terminus remote:wp my-site.env -- search-replace 'abc' ' ' --dry-run

# Don't put `;` in the search and replace strings. Use regex
# Since regex is slow, add certain table to search and limit the columns
terminus remote:wp my-site.env -- search-replace '(<script src="https:\/\/abc\.xyz\.net\/sub\/123_456_\/a\.js\?pid=.+" type="text\/javascript"><\/script>)' ' ' wp_posts --include-columns=post_content --skip-columns=guid  --dry-run --regex

# Bash script: Search/replace production to development url (multisite compatible)

#!/bin/bash
if $(wp --url=http://example.com core is-installed --network); then
    wp search-replace --url=http://example.com 'http://example.com' 'http://example.dev' --recurse-objects --network --skip-columns=guid --skip-tables=wp_users
else
    wp search-replace 'http://example.com' 'http://example.dev' --recurse-objects --skip-columns=guid --skip-tables=wp_users
fi

3.10.18. taxonomy

# default taxonomies are category and post_tag
wp taxonomy list

wp taxonomy get mytax_slug

3.10.19. term

wp term list category

wp term list mytax_slug

wp term delete category 15
wp term delete category apple --by=slug

# delete all terms for a taxonomy post_tag
wp term list post_tag --field=term_id | xargs wp term delete post_tag

3.10.20. eval

It doesn't work on Pantheon's Terminus 

wp eval "echo rand(); echo 'abc';" --skip-wordpress --allow-root
wp eval 'print(phpinfo());' --skip-wordpress --allow-root
wp eval 'print(phpinfo());' --skip-wordpress --allow-root | grep max_input_vars

3.10.21. config list

3.10.22. embed wp-cli:embed

3.10.23. Common Issues

All common issues

3.10.23.1. Can't connect to database

https://make.wordpress.org/cli/handbook/common-issues/#error-cant-connect-to-the-database

Make sure this line is exactly like this and wp-config.php shouldn't run any WP functions but only define constants 

require_once(ABSPATH . 'wp-settings.php');

Latest version of WordPress Docker is PHP 7 and hence has some extensions removed like mysql_connect. Make sure your PHP environment is the same for WordPress and wp-cli.

3.10.23.2. Out of memory
php -i | grep php.ini

# /usr/local/etc/php/conf.d/myphp.ini
memory_limit=512M

# then apache:restart

# Or you can run this to temporarily change the memory limit
php -d memory_limit=512M "$(which wp)" package install <package-name>
3.10.23.3. Allow root wp-cli:allow-root

This is for interactive shell 

alias wp='wp --allow-root'

Create a bash script wrapper for already started processes 

#!/bin/bash
exec /usr/local/bin/wp2 "$@" --allow-root

3.11. URL, Current URL, and slug wp:current url

Current URL on your website php:parse_url 

global $wp;

echo $wp->request; // path e.g. /path/to/page

echo home_url( $wp->request ); // works for pretty permalink

echo add_query_arg( $wp->query_vars, home_url( $wp->request )); // regardless of permalink setting

// Old method and may not work
// $current_url = home_url(add_query_arg(array(), $wp->request));
// echo esc_url($current_url);

$url = 'http://username:password@hostname:9090/path?arg=value#anchor';
$_url_array = parse_url($url); 
// keys are scheme, host, port, user, pass, path, query (after ?), fragment (after #)

if ($_url_array !== false && !is_null($_url_array['host']) {}

// Slug

$_post = get_post();
if (!empty($_post) && $_post->post_name == 'your-slug') {
  // do something
}

// Site URL

// http://google.com without trailing slash
get_site_url();

// Custom Link
echo '<a href="'.esc_url( 'somelink' ).'" target="_blank">'.__('some text', 'textdomain').'</a>';

get_permalink( int|WP_Post $post, bool $leavename = false )

  • Depends on post type, it can call other functions
    • get_page_link()
    • get_attachment_link()
    • get_post_link()
  • pre_post_link post_link_category post_link
// Get term URL
echo get_term_link( 'term-slug-or-term-obj-or-term-id', 'taxonomy-slug' );

3.12. WP_Post Post Object wp:post

$p = get_post();
echo $p->ID; // int

3.12.1. has_term wp:f:has_term

// has_term( $term = '', $taxonomy = '', $post = null )
// $term :: name/term_id/slug or array of them

3.13. WP_User, user object wp:user

Public properties ID (int) - the user's ID. caps (array) - the individual capabilities the user has been given. cap_key (string) - roles (array) - the roles the user is part of. allcaps (array) - all capabilities the user has, including individual and role based. first_name (string) - first name of the user. last_name (string) - last name of the user.

Public Methods has_cap :: has role name or capability 

if ( is_user_logged_in() ) {
  $user = wp_get_current_user();
  // $user->ID public properties

  if ($user->has_cap('administrator')) {
    // user is admin
  }
}

3.14. WP Custom Fields wp:Custom_Fields

3.14.1. Get Custom Fields

Normal 

print_r(get_post_custom_keys()); // Get all keys as an array
print_r(get_post_custom()); // Get keys and values as an array
print_r(get_post_custom_values('fieldkey')); // Get values as an array

// get all meta
$meta = get_post_meta(get_the_ID());
global $post;
echo get_post_meta($post->ID, 'my-ad', true);

Plugin is different :: wp:plugin:types

3.14.2. Custom Fields in WP_Meta_Query

Custom fields created by wp:plugin:acf can use WP_Meta_Query as normal.

Custome fields creasted by wp:plugin:types, like a set of checkboxes, in WP_Query, you should use the LIKE operator. Refer to WP Types Custom Field Checkbox and WP Meta Query

3.14.3. Create Custom Fields for Custom Post Type

Use wp:plugin:acf

3.15. WP_Query The Loop wp:The_Loop wp:query:main

  • The main query/loop is based on the URL request and is processed before templates are loaded
  • The wp:conditional tags use the main query/loop

wp:query:secondary

  • The secondary query/loop is new WP_Query in theme template or plugin files
  • It's also called The Loop
  • wp:template tags use the secondary query/loop
  • By default, The Loop uses the main query's current post to set global $post
  • new WP_Query() and the_post (called in have_posts()) modify The Loop and hence global $post is modified
  • After these new WP_Query() and the_post (Loop functions) are used, run wp:f:wp_reset_postdata to reset global $post to the original
  • new WP_Query() doesn't change wp:query:main
  • Refer to wp:global:wp_query for raw query

3.15.1. Syntax

<?php 
 $query = new WP_Query('cat=-3,-8'); 
 // Use query_posts($query_array | $string) instead of creating a new WP_Query then you don't have to prefix $query for methods
 // But query_posts change the global $wp_query 
?>

<?php if ( $query->have_posts() ) : while ( $query->have_posts() ) : $query->the_post(); ?>
 <?php //  set the global $post to individual one in $query ?>
 <div><?php the_content(); ?></div>

 <?php if ( has_post_thumbnail() ): ?>
  <img src="<?php the_post_thumbnail_url('large'); ?>">
  <?php the_post_thumbnail(); ?>
 <?php endif; ?>

<?php endwhile; ?>
<?php wp_reset_postdata(); // set back global $post to original ?>
<?php endif; ?>

<?php $query->rewind_posts(); // in order to loop $query again ?>

3.15.2. WP_Query

3.15.2.1. Basics
Generator
https://generatewp.com/wp_query/
Class Reference
https://codex.wordpress.org/Class_Reference/WP_Query
(no term)
Use objects in order
Class methods and properties
wp-includes/class-wp-query.php
Debug
wp:debug:wp_query
Public vs Private query vars
Refer to wp:filter:query_vars
$args = array(
    'post_type' => array('editorial', 'blog'), // public string|array d:'post'. Refer to wp:f:register_post_type
    // 'any' can be used
    // If 'tax_query' is used, default is changed to 'any'

    'post_status' => array('publish'), // string|array d:'publish'. Refer to wp:f:register_post_status
    // If user is logged in, 'private' is added.
    // Custom post statuses are added.
    // If is in admin, protected statuses are added: future, draft, pending
    // 'any' can be used

    'posts_per_page' => -1, // int. -1 shows all posts. Doesn't work in feed (Default 10). Use wp:filter:post_limits
    'post_mime_type' => 'image', // default:not set. string|array. e.g. `image/jpeg` | `image` (for all images)
    // $all_mimes = get_allowed_mime_types();
    // $unsupported_mimes = ['image/jpeg', 'image/gif', 'image/png', 'image/bmp', 'image/tiff', 'image/x-icon' ];
    // $accepted_mimes = array_diff( $all_mimes, $unsupported_mimes);
    // 'post_mime_type' => $accepted_mimes

    // Tag Parameters
    'tag' => 'tag-slug', // public. Can be 'bread,baking' for one of or 'bread+backing' for all
    'tag_id' => 123, // public
    'tag__and' => [123,321], // private
    'tag__in' => [123,321], // private
    'tag__not_in' => [123,321], // private
    'tag_slug__and' => ['tag1-slug','tag2-slug'], // private
    'tag_slug__in' => ['tag1-slug','tag2-slug'], // private

    // Category Parameters
    'cat' => '123', // public. Can be '2,6,17,38' for one of or '-12,-34,-56' for except
    'category_name' => 'category-slug', // public. Can be 'staff,news' for one of or 'staff+news' for all
    'category__and' => [123,321], // private
    'category__in' => [123,321], // private
    'category__not_in' => [123,321], // private

    'meta_query' => $meta_query, // $meta_query is a nested array parsed by WP_Meta_Query

    // Taxonomy Parameters
    // 'tax' => '', deprecated
    'tax_query' => $tax_query, // refer to WP Tax Query

    // Refer to WP Order and Orderby
    'order' => 'DESC', // public d:'ASC' 
    'orderby' => 'date', // public d:'date' which is post_date. Others are publish_date, title

    'suppress_filters' => false, // default. When true, filters will not run
);

$query = new WP_Query($args);

// or
// $q = new WP_Query;
// $q->query($args);
3.15.2.2. Get child posts
$args = [
    'post_type' => 'articles',
    'post_parent' => get_the_ID(),
    'posts_per_page' => -1,
    'orderby' => 'menu_order'
];
$children = new WP_Query($args);
3.15.2.3. Get attached images
$images = get_attached_media( 'image' );
// get_attached_media( string $type, int|WP_Post $post = 0 )
// $type :: mime type
// $post :: default is global $post
// return array

foreach ($images as $image) { ?>
    <img src="<?php echo wp_get_attachment_image_src($image->ID,'full'); ?>" />
<?php
}
3.15.2.4. Loop the main loop
<?php if (have_posts()) : while (have_posts()) : the_post(); ?>

    <div <?php post_class(); ?> id="post-<?php the_ID(); ?>">
        <h1><a href="<?php the_permalink(); ?>"><?php the_title(); ?></a></h1>
        <?php the_content(); ?>
    </div>

<?php endwhile; ?>

    <div class="navigation">
        <div class="next-posts"><?php next_posts_link(); ?></div>
        <div class="prev-posts"><?php previous_posts_link(); ?></div>
    </div>

<?php else : ?>

    <div <?php post_class(); ?> id="post-<?php the_ID(); ?>">
        <h1>Not Found</h1>
    </div>

<?php endif; ?>
3.15.2.5. Loop with query_posts() wp:f:query_posts
<?php global $query_string; // required
$posts = query_posts($query_string.'&cat=-9'); // exclude category 9
// $posts = query_posts($query_string.'&posts_per_page=3&cat=-6,-9&order=ASC');
?>

<?php // DEFAULT LOOP GOES HERE ?>

<?php wp_reset_query(); // reset the query ?>
3.15.2.6. Loop with new WP_Query()
<?php $custom_query = new WP_Query('cat=-9'); // exclude category 9
while($custom_query->have_posts()) :
    $custom_query->the_post(); // global $post is modified ?>

    <div <?php post_class(); ?> id="post-<?php the_ID(); ?>">
        <h1><a href="<?php the_permalink(); ?>"><?php the_title(); ?></a></h1>
        <?php the_content(); ?>
    </div>

<?php
endwhile;
wp_reset_postdata(); // reset the query

Array join 2 query results

  • Although php:array_merge is used, since 2 arrays have numeric keys, it's actually array join
<?php
$args = array(
    'post_type'      => 'speaker',
    'posts_per_page' => - 1,
    'orderby'        => 'meta_value',
    'meta_key'       => 'speaker_last_name',
    'order'          => 'ASC'
);

$tax_query_keynote = array(
    array(
        'taxonomy' => 'speaker_type',
        'field'    => 'slug',
        'terms'    => 'keynote-speakers',
    )
);

$tax_query_not_keynote = array(
    array(
        'taxonomy' => 'speaker_type',
        'field'    => 'slug',
        'terms'    => 'keynote-speakers',
        'operator' => 'NOT IN'
    )
);

$args_not_keynote = $args;
$args['tax_query'] =  $tax_query_keynote;
$args_not_keynote['tax_query'] = $tax_query_not_keynote;

$speakers_keynote = new WP_Query($args);
$speakers_not_keynote = new WP_Query($args_not_keynote);
$speakers = new WP_Query;

$speakers->posts = array_merge($speakers_keynote->posts, $speakers_not_keynote->posts);
$speakers->post_count = $speakers_keynote->post_count + $speakers_not_keynote->post_count;

if ($speakers->have_posts()) {
    while ($speakers->have_posts()) {
        $speakers->the_post();
        echo get_the_title();
        the_title(); // the_*() functions can be used as global $post is set
    }
}
3.15.2.7. Loop with get_posts()
  • Use new WP_Query(), the same $args
  • Return an array of WP_Post object wp:post
  • most filters are not run
  • It's good when new WP_Query($args) does not work because of other plugins/theme have extra filters that mess up WP_Query
  • Very safe to use
global $post; // required
$args = array('category' => -9); // exclude category 9
$custom_posts = get_posts($args);

// have to use $post in foreach
foreach($custom_posts as $post) {
    setup_postdata($post); // global $post is set and modified
  echo get_the_title();
}
wp_reset_postdata(); // important

// loop without modifying global $post
foreach ($custom_posts as $p) {
    the_title( $speaker->ID );
}
// don't need to reset postdata
3.15.2.8. Loop outside a WordPress Loop wp:loop:outside
if ( have_posts() ) {
    $items = [];

    while ( have_posts() ) {
        the_post();
        $items[] = new Agenda(
            get_the_ID(),
            get_field( 'agenda_date' ),
            get_field( 'agenda_start_time' ),
            get_field( 'agenda_end_time' ),
            get_permalink()
        );
    }

    // loop not using ->the_post
    foreach ($items as $item) {
        // some template functions you can use as it is
        echo get_the_title( $item->id );

        // for excerpt, use this
        $excerpt = apply_filters('the_excerpt', get_post_field('post_excerpt', $item->id));
    }

}
3.15.2.9. Loop inside loop, Loop Stack wp:loop:stack
if ( have_posts() ) {
    // main loop for single speaker content type
    while ( have_posts() ) {
        the_post(); // global $post is set and modified;

        global $post;
        $bk_speaker_post = clone $post; // clone the single speaker post

        the_title(); // the main speaker title. `the_*()` always uses the global $post

        // ACF field contains agenda
        $sessions = get_field( 'speaker_agenda', false, false );

        // loop the main speaker's agendas
        if ( $sessions ) {
            $sessions_sorted = new WP_Query( [
                'post_type'      => 'agenda',
                'posts_per_page' => - 1,
                'posts__in'      => $sessions
            ] );

            // loop agenda's speakers
            if ( $sessions_sorted ) {
                while ( $sessions_sorted->have_posts() ) {
                    $sessions_sorted->the_post(); // global $post is set and modified

                    global $post;
                    $bk_agenda_post = clone $post;

                    the_title(); // agenda title
                    echo get_the_title(); // agenda title
                    echo get_the_title( $bk_speaker_post->ID ); // speaker title!

                    // reverse query agenda to list speakers of this agenda
                    $agenda_speakers = get_posts( [
                        'post_type'  => 'speaker',
                        'meta_query' => [
                            [
                                'key'     => 'speaker_agenda',
                                'value'   => '"' . get_the_ID() . '"',
                                // matches exactly "123", not just 123. This prevents a match for "1234"
                                'compare' => 'LIKE'
                            ]
                        ]
                    ] );
                    foreach ( $agenda_speakers as $speaker ) {
                        // global $post is not modified and it's the agenda
                        $speakers_output[] = sprintf( '<a href="%s">%s</a>', get_permalink( $speaker->ID ), get_the_title( $speaker->ID ) );
                    }

                    // since global $post is not changed, it is still the agenda post

                    the_title(); // agenda title! not speaker title
                    echo get_the_title(); // agenda title
                    echo get_the_title( $bk_speaker_post->ID ); // the main speaker title!

                    // let's create another loop but now we change the global $post
                    // $post is global $post
                    foreach ( $agenda_speakers as $post ) {
                        setup_postdata( $post ); // $post is global $post
                        $speakers_output[] = sprintf( '<a href="%s">%s</a>', get_permalink(), get_the_title() );
                    }

                    the_title(); // the last speaker title of the current agenda! not agenda title as before
                    echo get_the_title(); // the last speaker title of the current agenda!
                    echo get_the_title( $bk_speaker_post->ID ); // the main speaker title!
                    echo get_the_title( $bk_agenda_post->ID ); // the current agenda title

                }
            }
            // loop agenda's speakers.
        }
        // loop the main speaker's agendas.
    }
    // loop the main speaker
}
3.15.2.10. is_*() methods wp:wp_query:m:is_*
is_home()
is_front_page()
Site Front Page. I use this one
(no term)
is_single( $post = '' )
is_singular( $post_types = '' )
string|array
is_post_type_archive()
is_post_type_archive('yourposttype')
If a custom tax archive page (taxonomy and/or a term) is being displayed
is_tax( $taxonomy= '' , $term = '') e.g. is_tax( 'my_tax' ) is_tax( 'my_tax', 'my_term' )
(no term)
is_main_query()
3.15.2.11. found_posts
3.15.2.12. post_count
  • Respect posts_per_page
3.15.2.13. current_post
Loop has not started or loop has just started
-1
(no term)
First item in a loop is zero
(no term)

Last post in a loop 

global $wp_query;

if ( ($wp_query->current_post + 1) == ($wp_query->post_count) ) {
    // last post
}
3.15.2.14. in_the_loop Whether posts are in a loop
3.15.2.15. post Current post
3.15.2.16. posts List of posts
3.15.2.17. query Query vars set by the user
3.15.2.18. query_vars Query vars, after parsing wp:wp_query:query_vars
3.15.2.19. have_posts()
  • At loop end
    • wp:action:loop_end
    • $this->current_post = -1; $this->post = $this->posts[0];
  • wp:action:loop_no_results
3.15.2.20. the_post()
  • $this->in_the_loop = true;
  • $this->next_post()
  • $this->setup_postdata( $post )
3.15.2.21. query( $query ) wp:wp_query:m:query
$query
array of post objects or post IDs
(no term)
$q->init();
(no term)
$q->query = $this->query_vars = wp_parse_args( $query )
(no term)
return wp:wp_query:m:get_posts
3.15.2.22. parse_query( $query = '' ) wp:wp_query:m:parse_query
3.15.2.23. get_posts() wp:wp_query:m:get_posts
  • Abstract
    parse_query()
    wp:wp_query:m:parse_query
    (no term)
    wp:action:pre_get_posts
    wp:filter:posts_search
    clause WHERE for Search Result SQL
    wp:filter:posts_search_orderby
    clause ORDER BY for Search Result SQL
    wp:filter:posts_where
    (no term)
    wp:filter:posts_join
    (no term)
    wp:filter:comment_feed_join
    (no term)
    wp:filter:comment_feed_where
    (no term)
    wp:filter:comment_feed_groupby
    (no term)
    wp:filter:comment_feed_orderby
    (no term)
    wp:filter:comment_feed_limits
    (no term)
    wp:filter:posts_where_paged
    (no term)
    wp:filter:posts_groupby
    (no term)
    wp:filter:posts_join_paged
    (no term)
    wp:filter:posts_orderby
    (no term)
    wp:filter:posts_distinct
    (no term)
    wp:filter:post_limits
    (no term)
    wp:filter:posts_fields
    (no term)
    wp:filter:posts_clauses
    wp:action:posts_selection
    for caching plugins
    wp:filter:posts_where_request
    for caching plugins
    wp:filter:posts_groupby_request
    for caching plugins
    wp:filter:posts_join_request
    for caching plugins
    wp:filter:posts_orderby_request
    for caching plugins
    wp:filter:posts_distinct_request
    for caching plugins
    wp:filter:posts_fields_request
    for caching plugins
    wp:filter:post_limits_request
    for caching plugins
    wp:filter:posts_clauses_request
    for caching plugins
    wp:filter:posts_request
    completed SQL query
    (no term)
    wp:filter:posts_pre_query
    (no term)
    wp:filter:split_the_query
    • wp:filter:posts_request_ids
    (no term)
    wp:filter:posts_results
    (no term)
    wp:filter:comment_feed_join
    (no term)
    wp:filter:comment_feed_where
    (no term)
    wp:filter:comment_feed_groupby
    (no term)
    wp:filter:comment_feed_orderby
    (no term)
    wp:filter:comment_feed_limits
    (no term)
    wp:filter:the_preview
    (no term)
    wp:filter:the_posts
    (no term)
    Other filters called in other WP_Query functions
    pre_option_posts_per_rss
    wp:filter:pre_option_*
    (no term)
    wp:filter:found_posts

3.15.3. WP_Meta_Query

https://codex.wordpress.org/Class_Reference/WP_Meta_Query https://rudrastyh.com/wordpress/meta_query.html

If only one condition is needed, use these fields

meta_key
string
meta_value
string|array
meta_type
string
meta_compare
string
$rd_args = array(
    'meta_key' => 'show_on_homepage',
    'meta_value' => 'on',
    'meta_compare' => '!='
);

$rd_query = new WP_Query( $rd_args );

Or use meta_query for nesting multiple conditions

3.15.3.1. meta_query Nesting

Nested: key1=value1 OR (key2=value2 AND key3=value3) 

$meta_args = array(
 'relation' => 'OR', // Optional. Default is 'AND'
 array(
   'key' => 'key1',
   'value' => 'value1',
   'compare' => '=', //default
 ),

 array(
   'relation' => 'AND',
   array(
     'key' => 'key2',
     'value' => 'value2',
     'compare' => '=',
   ),
   array(
     'key' => 'key3',
     'value' => 'value3',
     'compare' => '=',
   ),
 ),

);

$meta_query = new WP_Meta_Query($meta_args); // parse args array

To get posts that have featured images 

$args = array(
    'post_type'      => array( 'news', 'videos' ),
    'meta_query' => array(
        array('key'=> '_thumbnail_id')
    ),
    'posts_per_page' => 4,
);

$gamechangers = new WP_Query( $args );
3.15.3.2. Named sub-meta queries and multiple orderby arguments
  • The following requires all sort columns have values
  • The best to solve sort column might not have values is to make sure all posts have values for that sort column 
$q = new WP_Query( array(
    'meta_query' => array(
        'relation' => 'AND',
        'state_clause' => array(
            'key' => 'state',
            'value' => 'Wisconsin',
        ),
        'city_clause' => array(
            'key' => 'city',
            'compare' => 'EXISTS',
        ), 
    ),
    'orderby' => array( 
        'city_clause' => 'ASC', // 'key' is used to order
        'state_clause' => 'DESC',
    ),
) );
3.15.3.3. Review full query
$q_args = array(...);
$meta_query = new WP_Meta_Query();
$meta_query->parse_query_vars( $q_args );
$mq_sql = $meta_query->get_sql(
    'post',  // post, comment, user
    $wpdb->posts, // the table to look for rows
    'ID', // for posts, it's ID, for comments it's comment_ID, for users is ID
    null // $context (obj) optional
);

// return
array (size=2)
  'join' => string ' INNER JOIN wp_postmeta ON (wp_posts.ID = wp_postmeta.post_id)' (length=62)
  'where' => string ' AND (wp_postmeta.meta_key = 'some_key' )' (length=40)
3.15.3.4. Operators: compare and type

compare

  • =, !=
  • >, >=, <, <=
  • LIKE, NOT LIKE
  • IN, NOT IN
  • BETWEEN, NOT BETWEEN
  • EXISTS, NOT EXISTS
  • mysql:regex
  • RLIKE (synonym of REGEXP)

type In MySQL, it means CAST(the field, to a type)

  • NUMERIC
  • BINARY
  • CHAR
  • works with 'compare' value BETWEEN only if the date is stored as YYYY-MM-DD
  • DATETIME
  • DECIMAL
  • MySQL Integer
  • mysql:datatype:datetime

WP Types Custom Field Checkbox and WP Meta Query Say the wpcf-newsletters custom field created by WP Types has 2 checkboxes are selected: enligne and pw. The following 2 strings will appear in the serialized json string as value in WP_Meta_Query 

a:1:{i:0;s:7:"enligne";}
a:1:{i:0;s:2:"pw";}

Select the posts that have checkbox enligne checked. 

$meta_query = array(
 array(
  'key' => 'wpcf-newsletters',
  'value' => 'a:1:{i:0;s:7:"enligne";}',
  'compare' => 'LIKE',
 ),
);

3.15.4. WP Tax Query

$tax_query = array(
    'relation' => 'AND', // Optional 
    array(),
    array(
        'taxonomy' => '',
        'field' => 'term_id', // string :: term_id (default), name, slug or term_taxonomy_id
        'terms' => 'bob', // int/string/array :: multiple terms
        'operator' => 'IN', // string :: IN (default), NOT IN, AND, EXISTS, NOT EXISTS
    ),
);

3.15.5. WP Order and Orderby

3.15.5.1. order
  • string or array to match orderby
  • default
  • ASC
3.15.5.2. orderby

3.15.6. WP_Term_Query wp:WP_Term_Query

// get parent term by slug
$sponsor_type = get_term_by( 'slug', 'sponsors', 'sponsor_type' );
if ( ! $sponsor_type ) {
    return;
}

// get child terms and sort by ACF field sponsorship_type_order
$args          = [
    'taxonomy'   => 'sponsor_type',
    'parent'     => $sponsor_type->term_id,
    'order'      => 'sponsorship_type_order',
    'orderby'    => 'ASC',
    'hide_empty' => true, // default true. false to return terms that have no post associated with it
];
$sponsor_types = new WP_Term_Query( $args );

foreach ( $sponsor_types->terms as $type_term ) {
    // get posts that has this term
    $args             = [
        'post_type'      => 'sponsors',
        'post_status'    => 'publish',
        'posts_per_page' => - 1,
        'tax_query'      => [
            [
                'taxonomy' => 'sponsor_type',
                'field'    => 'slug',
                'terms'    => $type_term->name
            ]
        ],
        'orderby'        => 'title',
    ];
    $sponsors_by_type = new WP_Query( $args );
    if ( $sponsors_by_type->have_posts() ) : ?>
        <h2 class="mb-5"><?php echo $type_term->name; ?></h2>
        <?php
        while ( $sponsors_by_type->have_posts() ) : $sponsors_by_type->the_post(); ?>
            <article class="row mb-3">
                <div class="col-md-3 align-self-center">
                    <a href="<?php the_permalink() ?>">
                        <?php if ( has_post_thumbnail() ) {
                            the_post_thumbnail( 'thumbnail', [
                                'class' => 'd-block img-fluid mx-auto mb-3',
                                'alt'   => esc_attr( get_the_title() ),
                            ] );
                        } else {
                            echo '<p class="h3 text-center">' . get_the_title() . '</p>';
                        }
                        ?>
                    </a>
                </div>
                <div class="col-md-9">
                    <?php the_content(); ?>
                </div>
            </article>
        <?php
        endwhile;
        wp_reset_postdata();
        ?>
    <?php
    endif;
}

3.15.7. Review Raw Query

$args=[];
$q = new WP_Query($args);
echo "<pre>";
print_r($q->request);
echo "</pre>";

SELECT wp_posts.*
FROM wp_posts
  -- tax_query uses LEFT JOIN
  LEFT JOIN wp_term_relationships ON (wp_posts.ID = wp_term_relationships.object_id)
  -- meta_query by default uses INNER JOIN
  INNER JOIN wp_postmeta ON ( wp_posts.ID = wp_postmeta.post_id )
  INNER JOIN wp_postmeta AS mt1 ON ( wp_posts.ID = mt1.post_id )
  -- meta_query when there's `NOT EXISTS` then all INNER JOIN are changed to LEFT JOIN
  --LEFT JOIN wp_postmeta AS mt1 ON ( wp_posts.ID = mt1.post_id AND mt1.meta_key= 'custom_rder')

WHERE 1=1  AND (
    wp_term_relationships.term_taxonomy_id IN (16896)
  )
  AND (
  wp_postmeta.meta_key = 'speaker_last_name'
    AND
  mt1.meta_key = 'custom_order'
  )
  AND wp_posts.post_type = 'speaker'
  AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'acf-disabled')
GROUP BY wp_posts.ID
ORDER BY CAST(mt1.meta_value AS CHAR) ASC, CAST(wp_postmeta.meta_value AS CHAR) ASC

3.16. Functions

3.16.1. Plugin API wp-includes/plugin.php wp:api:plugin

3.16.1.1. add_filter, apply_filters, apply_filters_ref_array wp:add_filter

Syntax, remove_filter

  • Modify internal data / function returned
  • For example, WP_Query gets internal settings for building a query
    • You can temporary add_filter to change the internal settings, then build a WP_Query, later remove_filter
    • so that you only change the internal setting for that query building without affecting other usage
add_filter( 
 'tagname', 
 'lili_funct',
  10, // int. Priority, lower number sooner
  2, // int. Accepted args, default is 1! Be careful
);

apply_filters( 'tagname', $result, $var1);

// callback should take 2 variables
function lili_funct($result, $var1) {
 $new = $result . 'abc';
 return $new;
}

remove_filter(
  'tagname',
  'lili_funct',
  10 // priority, must match the add_filter priority, default is 10
);

function remove_filter_the_title() {
    remove_filter('the_title', 'before_post_title', 10);
}
add_action( 'wp_head', 'remove_filter_the_title', 1 );

function add_filter_the_title() {
    add_filter('the_title', 'before_post_title', 10, 2);
}
add_action( 'wp_head', 'add_filter_the_title', 3 );

Class method 

// Static class method
add_filter( 'media_upload_newtab', array( 'My_Class', 'media_upload_callback' ) );

// Instance method
add_filter( 'media_upload_newtab', array( $this, 'media_upload_callback' ) );

// Anonymous function
add_filter( 'the_title', function( $title ) { return '<strong>' . $title . '</strong>'; } );
3.16.1.2. add_action, do_action, do_action_ref_array wp:add_action

Event based 

add_action( string $tag,
  string $function_name,
  int $priority = 10,
  int $accepted_args = 1 // number of args the function accepts
);

// add_action( 'atag' is called by do_action( 'atag' or do_action_ref_array( 'atag'

// child theme
add_action( 'abc', 'xyz' );
// parent theme
add_action( 'abc', 'abc' );

// xyz and abc will run in order
do_action( 'abc' );

// child theme
remove_action( 'abc', 'ijk' );

// parent theme
add_action( 'abc', 'ijk');

// ijk is added and then removed, so ijk will not run
do_action( 'abc' );
  1. Action sequence wp:action:loading sequence
3.16.1.3. register_activation_hook wp:f:register_activation_hook
  • register_activation_hook( string $file, callable $function)
  • register_deactivation_hook
// run something after a plugin is activated
// usually the code is added in wp-content/plugins/myplugin/myplugin.php, but it can be anywhere

/** wp-content/plugins/myplugin/myplugin.php
 * Register Task post type.
 */
require_once plugin_dir_path( __FILE__ ) . 'includes/posttypes.php';
register_activation_hook( __FILE__, 'taskbook_rewrite_flush' );

/** wp-content/plugins/myplugin/includes/posttypes.php
 * Register Task post type.
 */
function taskbook_cpt_init() {
        register_post_type( 'task', $args );
}
add_action( 'init', 'taskbook_cpt_init' );
function taskbook_rewrite_flush() {
    // First, we "add" the custom post type via the above written function.
    // Note: "add" is written with quotes, as CPTs don't get added to the DB,
    // They are only referenced in the post_type column with a post entry, 
    // when you add a post of this CPT.
    taskbook_cpt_init();

    // ATTENTION: This is *only* done during plugin activation hook in this example!
    // You should *NEVER EVER* do this on every page load!!
    flush_rewrite_rules();
}

3.16.2. Transients API - set_transient vs wp_cache_set wp:cache

WordPress cache API in database. DELETE FROM wp_options WHERE option_name LIKE ('%\_transient\_%')

WP object cache stores objects and primitives but not in a persistent manner by default. This means caching happens in memory and it only lives for the request's lifetime cycle (e.g. one pageload). Redis plugin can make it persistent.

https://codex.wordpress.org/Class_Reference/WP_Object_Cache

Transient API saves variables, arrays, objects tied with an expiration date on db and has persistent object caching. However when cached objects expire, they remain on db which needs to be pruned every once in a while.

WP detects if persistent object cache, if so, bypass Transients API and route to WP object cache. 

// wp_cache_get( int|string $key, string $group = '', bool $force = false, bool $found = null )
// wp_cache_set( int|string $key, mixed $data, string $group = '', int $expire )
$result = wp_cache_get( 'my_result', 'my-group' );
if ( false === $result ) {
    $result = $wpdb->get_results( $query );
    wp_cache_set( 'my_result', $result, 'my-group', 600 ); // expire after 5 mins
} 
// Do something with $result;

3.16.3. Formatting wp-includes/formatting.php

3.16.3.1. wp_strip_all_tags( $string, $remove_breaks = false )
function wp_strip_all_tags($string, $remove_breaks = false) {
  $string = preg_replace( '@<(script|style)[^>]*?>.*?</\\1>@si', '', $string );
  $string = strip_tags($string);

  if ( $remove_breaks )
    $string = preg_replace('/[\r\n\t ]+/', ' ', $string);

  return trim( $string );
}
3.16.3.2. wp_trim_words( $text, $num_words = 55, $more = null ) wp:ellipsis
  • Strip HTML and PHP tags and truncate string in words
function wp_trim_words( $text, $num_words = 55, $more = null ) {
  if ( null === $more ) {
    $more = __( '&hellip;' );
  }

  $original_text = $text;
  $text = wp_strip_all_tags( $text );

  /*
   * translators: If your word count is based on single characters (e.g. East Asian characters),
   * enter 'characters_excluding_spaces' or 'characters_including_spaces'. Otherwise, enter 'words'.
   * Do not translate into your own language.
   */
  if ( strpos( _x( 'words', 'Word count type. Do not translate!' ), 'characters' ) === 0 && preg_match( '/^utf\-?8$/i', get_option( 'blog_charset' ) ) ) {
    $text = trim( preg_replace( "/[\n\r\t ]+/", ' ', $text ), ' ' );
    preg_match_all( '/./u', $text, $words_array );
    $words_array = array_slice( $words_array[0], 0, $num_words + 1 );
    $sep = '';
  } else {
    $words_array = preg_split( "/[\n\r\t ]+/", $text, $num_words + 1, PREG_SPLIT_NO_EMPTY );
    $sep = ' ';
  }

  if ( count( $words_array ) > $num_words ) {
    array_pop( $words_array );
    $text = implode( $sep, $words_array );
    $text = $text . $more;
  } else {
    $text = implode( $sep, $words_array );
  }

  /**
   * Filters the text content after words have been trimmed.
   *
   * @since 3.3.0
   *
   * @param string $text          The trimmed text.
   * @param int    $num_words     The number of words to trim the text to. Default 55.
   * @param string $more          An optional string to append to the end of the trimmed text, e.g. &hellip;.
   * @param string $original_text The text before it was trimmed.
   */
  return apply_filters( 'wp_trim_words', $text, $num_words, $more, $original_text );
}
3.16.3.3. antispambot( $email_address, $hex_encoding = 0 ) : string wp:f:antispambot

  • Encode email address to hex 

    function hide_email_addresses_sc($atts, $content = null) {
    // $atts['var1'], the following just sets a default if var1 attribute is not defined in shortcode
    $emailaddress_fields = shortcode_atts(array(
        'id' => '',
        'email' => ''
    ),$atts);
    $userid = $emailaddress_fields['id'];
    $e_mail = $emailaddress_fields['email'];
    if ($userid !=='' && $e_mail =='') {
        $emailaddress = get_the_author_meta('user_email',$userid);
        return '<a href="mailto:'.antispambot($emailaddress).'">'.antispambot($emailaddress).'</a>';
    }
    if ($userid =='' && $e_mail !=='') {
        return '<a href="mailto:'.antispambot($e_mail).'">'.antispambot($e_mail).'</a>';
    }
    else {
        return '';
    }
}
add_shortcode('hide-email','hide_email_addresses_sc');
3.16.3.4. wpautop( $pee, $br = true ) : string wp:f:wpautop

Turn off wpautop for specific pages 

function remove_p_on_pages() {
    if ( is_page() ) {
    remove_filter( 'the_content', 'wpautop' );
    remove_filter( 'the_excerpt', 'wpautop' );

    }
}
add_action( 'wp_head', 'remove_p_on_pages' );

3.16.4. Main API wp-includes/functions.php wp:api:main

3.16.4.1. Option API wp:api:option
3.16.4.2. wp_parse_args( $args, $defaults = '' ) wp:f:wp_parse_args

Merge user defined arguments into defaults array

3.16.4.3. __return_*()
  • __return_true()
  • __return_false()
  • __return_empty_array()
  • __return_null()
  • __return_empty_string()
3.16.4.4. wp_upload_dir

Return an array to describe location of wp upload directory 

array (
    'path' => '/var/www/html/wp-content/uploads/2018/09', // next upload will go to this path
    'url' => 'http://localhost:32006/wp-content/uploads/2018/09', // next upload 
    'subdir' => '/2018/09', 
    'basedir' => '/var/www/html/wp-content/uploads',
    'baseurl' => 'http://localhost:32006/wp-content/uploads',
    'error' => false
)
3.16.4.5. do_feed() wp:f:do_feed
  • do_feed()
    • Get feedname from get_query_var('feed') and do wp:action:do_feed_$feedname
    • Default added feed actions
      do_feed_rdf()
      wp-includes/feed-rdf.php
      do_feed_rss()
      wp-includes/feed-rss.php
      do_feed_rss2( $for_comments )
      wp-includes/feed-rss2-comments.php or feed-rss2.php
      do_feed_atom( $for_comments )
      wp-includes/feed-atom-comments.php or feed-atom.php

3.16.5. Translate wp:translate

Use tool:Poedit

3.16.5.1. Translate Functions wp-includes/l10n.php
$text
can't be a variable. And use single quotes instead of ""
(no term)
Use sprintf (return) or printf() (echo) for subsitutions. The msgid is "You have chosen the %s theme"
  1. Sample
    echo sprintf( __( 'You have chosen the %s theme.' ) , the_color() );

printf( __( 'You have chosen the %s theme.' ) , the_color() );

echo __('some text', 'textdomain'); // return
// 'textdomain' is optional but it has to be a string not a variable! Default it's 'default'

// _e() is the same as __() but echoes

// Plural or single
// _n( $single, $plural, $number, $domain = 'default' )
echo _n('There is a comment', 'There are comments', get_comments_number());

_nx( $single, $plural, $number, $context, $domain = 'default' );

// Context
echo _x('post a link', 'A link to the post', 'my-text-domain');
// 'A link to the post' matches msgctxt "A link to the post" in .po file.

// _ex is the same as _x but echoes

// return value for an HTML attribute 
printf( esc_attr__( 'View all posts filed under %1$s', 'my-text-domain' ), get_cat_name( 32 ) );

// echo value for an HTML attribute
esc_attr_e( $text, $domain = 'default' );

esc_attr_x( $text, $context, $domain = 'default' );

// return value for HTML output
esc_html__( $text, $domain = 'default' );

esc_html_e( $text, $domain = 'default' );

esc_html_x( $text, $context, $domain = 'default' );
3.16.5.2. Setup, .po and .mo files

Load textdomain in child theme which overwrites the textdomain defined in parent theme Do this in theme functions.php 

function smart_mag_child_theme_locale() {
  $_r = load_child_theme_textdomain( 'textdomain-name', get_stylesheet_directory() . '/languages' );
  // textdomain-name can be defined in parent theme or a new textdomain
  // if textdomain-name has been defined earlier, then this will overwrite

  // Check if textdomain inclusion is successful
  //if ( $_r ) var_dump( 'success!' );

  if ( is_admin() ) {
    // load another textdomain
  }
}
add_action( 'after_setup_theme', 'smart_mag_child_theme_locale' );

Usually translate-ready theme will provide a .pot file which is a template file for .po and .mo files. Open .pot in tool:Poedit, generate a new translation. What Poedit saves is 2 files: .po and .mo.

Open .po file later, make changes and save to overwrite .po and .mo files. File .pot is no longer needed.

Catalog > Properties > Translation properties Specifiy the language Plural Forms by default is 

nplurals=2; plural=n != 1;

Might need to specify the path in code for .po file. '.' means the current path. 

"X-Poedit-SearchPath-0: .\n"

Catalog > Properties > Sources keywords contain all the Wordpress translation functions 

__
_e
_n:1,2
_x:1,2c
_ex:1,2c

:1,2 indicates the function has 2 parts. 2c means the 2nd argument is a context/comment.

tool:Poedit can help you change most of the fields except that it can't specify context msgctxt.

For those situations, directly edit .po and open .po in Poedit and save to get the .mo file.

Some examples

Comments :: #. Author of the plugin/theme Due to emacs, ;; #. is #. in the following examples Reference :: #: woocommerce/loop/orderby.php:23 Due to emacs, ;; #: is #: in the following examples

Concatenate multiple lines 

#: 404.php:18
msgid ""
"We're sorry, but we can't find the page you were looking for. It's probably "
"some thing we've done wrong but now we know about it and we'll try to fix "
"it. In the meantime, try one of these options:"
msgstr ""

sprintf and printf subsitutions 

#: archive.php:101
msgid "Yearly Archives: %s"
msgstr ""

Context 

#: bbpress/auth-modal.php:69
msgctxt "bbPress"
msgid "Favorites"
msgstr ""

Complex context 

#: comments.php:53
msgid ""
"Logged in as <a href=\"%1$s\">%2$s</a>. <a href=\"%3$s\" title=\"Log out of "
"this account\">Log out?</a>"
msgstr ""

Poedit save fr_CA.po and fr_CA.mo files. Copy and place them into the right place and then on Wordpress Settings > General > Site Lanuage to match fr_CA

3.16.6. User Role & Capabilities API wp:api:role-capabilities wp:roles

  • https://codex.wordpress.org/Roles_and_Capabilities

  • Capabilities 

    $role = get_role('editor'); // WP_Role
$role->add_cap('edit_theme_options');

// later
if ( current_user_can( 'edit_theme_options' ) ) {}
    manage_options
    default admin only. Many plugins use this to determine if the user can make plugin specific changes
    edit_theme_options
    Add Admin Panel options for Appearance:
    • Widgets
    • Menus
    • Customize
    • Background
    • Header
    (no term)

    Post type capabilities 

      'capabilities' => array(
'edit_post'          => 'edit_book', 
'read_post'          => 'read_book', 
'delete_post'        => 'delete_book', 
'edit_posts'         => 'edit_books', 
'edit_others_posts'  => 'edit_others_books', 
'publish_posts'      => 'publish_books',       
'read_private_posts' => 'read_private_books', 
'create_posts'       => 'edit_books', 
  )

  [cap] => stdClass Object
  (
      // Meta capabilities

      [edit_post]        => "edit_{$capability_type}"
      [read_post]        => "read_{$capability_type}"
      [delete_post]      => "delete_{$capability_type}"

      // Primitive capabilities used outside of map_meta_cap():

      [edit_posts]       => "edit_{$capability_type}s"
      [edit_others_posts]    => "edit_others_{$capability_type}s"
      [publish_posts]        => "publish_{$capability_type}s"
      [read_private_posts]   => "read_private_{$capability_type}s"

      // Primitive capabilities used within map_meta_cap():

      [read]                   => "read",
      [delete_posts]           => "delete_{$capability_type}s"
      [delete_private_posts]   => "delete_private_{$capability_type}s"
      [delete_published_posts] => "delete_published_{$capability_type}s"
      [delete_others_posts]    => "delete_others_{$capability_type}s"
      [edit_private_posts]     => "edit_private_{$capability_type}s"
      [edit_published_posts]   => "edit_published_{$capability_type}s"
      [create_posts]           => "edit_{$capability_type}s"
  )
3.16.6.1. add_role( string $role, string $display_name, $capabilities = array() ) : WP_Role|null wp:f:add_role
$result = add_role(
    'basic_contributor',
    __( 'Basic Contributor' ),
    array(
        'read'         => true,  // true allows this capability
        'edit_posts'   => true,
        'delete_posts' => false, // Use false to explicitly deny
    )
);

if ( null !== $result ) {
    echo 'Yay! New role created!';
}
else {
    echo 'Oh... the basic_contributor role already exists.';
}

// Create a new role when a plugin is activated
function add_roles_on_plugin_activation() {
    add_role( 'custom_role', 'Custom Subscriber', array( 'read' => true, 'edit_posts' => true ) );
}
register_activation_hook( __FILE__, 'add_roles_on_plugin_activation' );

3.16.7. Query API wp:api:query

  • wrapper of glboal $wp_query e.g. is_* functions
  • query_posts( $query ) wp:f:query_posts
3.16.7.1. Conditional Tags wp:conditional tags
  1. is_page( int|string|array $page = '' )
    is_page(); // any page
is_page(42);
is_page( 'mypageslug' );
is_page( array(42, 'about-me', 'Page title' ) ); // either one of those id, slug or title
is_page( array(42, 54, 6) );
  2. is_page_template( string|array $template = '' )
    is_page_template(); // any page template
is_page_template( 'about.php' ); // another template 'page-templates/about.php'
  3. is_single( int|string|array $post = '' )
    • works for any post type except attachments and pages
    • Post ID, title, slug or array of such (one of the specified)
  4. is_singular( string|array $post_types = '' )
    $post_types
    Whether the query is for an existing single post of any post type. Or if $post_types is specified, additionally check if the query is for one of the Post Types specified
    is_singular();
is_singular( 'article' );
is_singular( array( 'newspaper', 'book' ) );
  5. is_admin_bar_showing()
3.16.7.2. wp_reset_postdata wp:f:wp_reset_postdata
  • Use it when custom query new WP_Query() is defined and used
  • After global $post is modified by wp:query:secondary, reset it to the original that was set by wp:query:main
<?php
// example args
$args = array( 'posts_per_page' => 3 );

// the query
$the_query = new WP_Query( $args );
?>

<?php if ( $the_query->have_posts() ) : ?>

    <!-- start of the loop -->
    <?php while ( $the_query->have_posts() ) : $the_query->the_post(); ?>
        <?php the_title(); ?>
        <?php the_excerpt(); ?>
    <?php endwhile; ?><!-- end of the loop -->

    <!-- put pagination functions here -->
    <?php wp_reset_postdata(); ?>

<?php else:  ?>

<p><?php _e( 'Sorry, no posts matched your criteria.' ); ?></p>

<?php endif; ?>
3.16.7.3. wp_reset_query wp:f:wp_reset_query
  • Should be called after query_posts()
  • $GLOBALS['wp_query'] = $GLOBALS['wp_the_query']; and wp_reset_postdata()

Avoid to use it. To alter main query parameters before it's made, use wp:action:pre_get_posts 

$args = array ( 'post_parent' => 5 );
query_posts( $args );

if ( have_posts() ):
    while ( have_posts() ) :
    the_post();

    // Do stuff with the post content.
    the_title();
    the_permalink(); // Etc.

    endwhile;
else:
    // Insert any content or load a template for no posts found.
endif;

wp_reset_query();

3.16.8. Theme functions wp-includes/theme.php

3.16.8.1. get_template_directory, get_template_directory_uri, get_stylesheet_directory_uri, get_stylesheet_uri
get_template_directory()
the filesystem path of the current parent theme
get_template_directory_uri
URI e.g. protocol://domain of the current parent theme
get_stylesheet_directory()
the filesystem path of the current child theme
get_stylesheet_directory_uri
URI of the stylesheet directory. grab child and then parent theme.
get_stylesheet_uri
URI of style.css
include( get_template_directory() . '/includes/myfile.php');
wp_enqueue_script( 'twentyfifteen-skip-link-focus-fix', get_template_directory_uri() . '/js/skip-link-focus-fix.js', array(), '20141010', true );
3.16.8.2. get_theme_mod wp:f:get_theme_mod
get_theme_mod( $name, $default = false )
3.16.8.3. add_theme_support, remove_theme_support, current_theme_supports wp:f:add_theme_support

wp:f:current_theme_supports wp:f:add_theme_support:post-thumbnails 

add_theme_support( 'post-thumbnails' );

// refer to wp:f:add_image_size

add_theme_support( 'automatic-feed-links' );
// adds
//<link rel="alternate" type="application/rss+xml" title="Website Name Feed" href="http://myweb.io/feed/" />
//<link rel="alternate" type="application/rss+xml" title="Website Name Comments Feed" href="http://myweb.io/comments/feed/" />
// If it's a page, add one more
// <link rel="alternate" type="application/rss+xml" title="Website Name Page Name Comments Feed" href="http://myweb.io/2016/07/20/page-name/feed/" />

add_theme_support( 'title-tag' ); // wp will provide <title> not hard coded in template file using wp_title()

add_theme_support( 'html5', array(
  'search-form',
  'comment-form',
  'comment-list',
  'gallery',
  'caption',
) );
// use the default html5 markup instead of the default html4 file

add_theme_support( 'post-formats', array(
  'aside',
  'image',
  'video',
  'quote',
  'link',
  'gallery',
  'audio',
) );
// wp provides a list of formats for admin users to choose to display a post
// refer to wp:f:has_post_format

add_theme_support( 'custom-logo', array(
        'width'       => 250,
        'height'      => 250,
        'flex-width'  => true,
) );
// provide a place to upload theme logo in wp-admin
// https://developer.wordpress.org/themes/functionality/custom-logo/
// refer to wp:custom logo:display

add_theme_support( 'customize-selective-refresh-widgets' );
// https://make.wordpress.org/core/2016/03/22/implementing-selective-refresh-support-for-widgets/

add_theme_support( 'starter-content', $starter_content );
// https://make.wordpress.org/core/2016/11/30/starter-content-for-themes-in-4-7/

//add_theme_support( 'custom-background' ); // simple enable
$defaults = array(
    'default-color'          => '',
    'default-image'          => '',
    'default-repeat'         => 'repeat',
    'default-position-x'     => 'left',
    'default-position-y'     => 'top',
    'default-size'           => 'auto',
    'default-attachment'     => 'scroll',
    'wp-head-callback'       => '_custom_background_cb',
    'admin-head-callback'    => '',
    'admin-preview-callback' => ''
);
add_theme_support( 'custom-background', $defaults );
// https://codex.wordpress.org/Custom_Backgrounds
3.16.8.4. add_editor_style( array|string $stylesheet = 'editor-style.css' )
  • Used in
  • Add stylesheets for TinyMCE
  • Stylesheet name or array thereof, relative to theme root
  • This function automatically adds another stylesheet with -rtl prefix, e.g. editor-style-rtl.css. If that file doesn’t exist, it is removed before adding the stylesheet(s) to TinyMCE. If an array of stylesheets is passed to add_editor_style(), RTL is only added for the first stylesheet.
  • Since version 3.4 the TinyMCE body has .rtl CSS class. It's a better option to use that class and add any RTL styles to the main stylesheet
add_editor_style( array( 'assets/css/editor-style.css', twentyseventeen_fonts_url() ) );
3.16.8.5. get_theme_root, get_theme_root_uri
$theme_root = get_theme_root();
$files_array = glob("$theme_root/*", GLOB_ONLYDIR);
echo "There are " . count($files_array) . " subdirectories in the " . $theme_root . " directory"; 
// There are 5 subdirectories in the /home/user/public_html/wp-content/themes directory.

echo get_theme_root_uri();
// http://www.wordpress.org/wp-content/themes

3.16.9. Template Tags wp:template tags

3.16.9.1. post-thumbnail-template.php wp:t:thumbnail
has_post_thumbnail( $post = null)
return bool
get_post_thumbnail_id( $post = null, $size = 'post-thumbnail', $attr = '' );
get thumbnail id
get_the_post_thumbnail_*()
can specify which post
  • wp_get_attachment_image_src is called
  • return html with img attributes
    srcset
    wp_calculate_image_srcset
    sizes
    wp_calculate_image_sizes
  • string|false
the_post_thumbnail_*
echo instead of return. calls get_the_post_thumbnail_*
the_post_thumbnail( $size, $attr = '' )
echo get_the_post_thumbnail( null, $size, $attr )
  • e.g. the_post_thumbnail( 'medium-large', array( 'class' => 'my-class') )
the_post_thumbnail_url( $size )
echo url
the_post_thumbnail_caption( $post = null )
echo thumbnail caption
wp_get_attachment_image_*()
wp_get_attachment_image_src is called
wp_get_attachment_image_src( $attachment_id, string|array $size = 'thumbnail', $icon = false )
just like the_post_thumbnail but returns an array [url, width, height, is_intermediate]. url is $thumb[0]
wp_get_attachment_image( $attachment_id, string|array $size = 'thumbnail', $icon = false, string|array $attr = '' )
return HTML img element or empty string on failure
wp_get_attachment_image_url( $attachment_id, $size = 'thumbnail', $icon = false )
return string|false Attachment URL or false if no image is available
wp_get_attachment_image_srcset( $attachment_id, $size = 'medium', $image_meta = null )
srcset attrbiute value string or false
wp_get_attachment_image_sizes( $attachment_id, $size = 'medium', $image_meta = null )
sizes attribute value string or false
wp_get_attachment_*()
wp_get_attachment_caption( $post_id = 0 )
$post_id default is global $post. Usually it's the attachment ID
wp_get_attachment_metadata( $attachment_id = 0, $unfiltered = false )
array or false
wp_get_attachment_url( $attachment_id = 0)
wp_get_attachment_thumb_file( $post_id = 0 )
wp_get_attachment_thumb_url( $post_id = 0)
Get alt
$alt = get_post_meta( get_post_thumbnail_id(), '_wp_attachment_image_alt', true);
(no term)
Refer to wp:f:add_image_size
NOTICE
If the thumbnail has no physical file that has corresponding size, the_post_thumbnail_url will get the original image file, the_post_thumbnail will use the original file with in src and apply width/height attributes in HTML to achieve the purpose.
'post-thumbnail'
default
array(100, 100)
Grab the biggest size in width and height image file possible.
array(100, 9999)
Grab the biggest size in width and height image file possible.
(no term)
Resize an image file on the fly using fly-dynamic-image-resizer
3.16.9.2. post-template.php wp:t:post
  • Only get_the_*( $post ) can pass a different post or post id to return values. Use esc_html_e() to echo
    • get_the_guid( $post = 0)

    • wp:f:get_the_title

      • if (! is_admin() )
      wp:filter:protected_title_format
      add Protected:
      wp:filter:private_title_format
      add Private:
      $post->post_title
      wp:filter:the_title
    • get_the_content( $more_link_text = null, $strip_teaser = false, $post = null )
    • get_the_excerpt( $post = null )
    • get_the_password_form( $post = 0 )
  • ID of current item or false
    the_ID()
    echo
  • the_*() always uses the global $post to echo
    the_title( $before = '', $after = '', $echo = true )
    return or echo $before . get_the_title() . $after
    the_title_attribute( string|array $args = '' )

    similar to above but tags are stripped and escaped for HTML attributes 

    $defaults = [
    'before' => '',
    'after' => '',
    'echo' => true,
    'post' => get_post(),
];
    the_content( $more_link_text = null, $strip_teaser = false)
    use wp:filter:the_content
    (no term)
    the_excerpt()
    When another post/post id is passed on
    $excerpt = apply_filters('the_excerpt', get_post_field('post_excerpt', $item->id));
  • get_post_class()
    • e.g. post-123 myposttype type-myposttype status-publish has-post-thumbnail
    • filter post_class($classes, $class, $post->ID)
  • wp_list_pages( $args = '' )
    • $args
      post_type
      default page. For CPT, wp_list_pages( array('post_type' => 'articles' ) )
      echo
      default 1, to echo. Set 0 to return html string
      title_li
      default __( 'Pages' ). Add a <li> as title before any child post elements
      child_of
      default 0 (return child posts of any parent posts). specify the parent post ID
      sort_column
      default 'menu_order, post_title'
  1. get_the_excerpt, the_excerpt
3.16.9.3. category-template.php wp:t:category
  • Refer to wp:post
  • display taxonomy terms in a list
  • wp_list_categories( $args = '')
    taxonomy
    default 'category'. For custom taxonomy, wp_list_categories( [ 'taxonomy' => 'mycats' ] )
    child_of
    default 0. Specify the parent term id
3.16.9.4. general-template wp:t:general
// In template
<?php wp_head(); ?>

// In plugin
add_action('wp_head', 'lili_wphead_dfp');
function lili_wphead_dfp() {
 // add inline script to header
 $o = "<script>...</script>";
 echo $o;
}

// Load with dependancy (jQuery is loaded)
function lili_wphead_dfp() {
 if ( wp_script_is( 'jquery', 'done' ) ) {
?>
<script type="text/javascript">
// jQuery code 
</script>
<?php
 }
}
  1. get_header, get_footer, get_sidebar, get_search_form, comments_template
    wp:f:get_header
    get_header( $name = null )
    wp:f:get_footer
    get_footer( $name = null )
    wp:f:get_sidebar
    get_sidebar( $name = null ) Refer to wp:f:dynamic_sidebar
    • $templates[] = "sidebar-{$name}.php"; $templates[] = 'sidebar.php';
    • locate_template( $templates, true );
    wp:f:get_search_form
    get_search_form( $args = array() )
    (no term)
    wp:f:comments_template
    (no term)

    These functions are to load template files

    System default load paths
    wp-includes/theme-compat/header.php, footer.php, sidebar.php
    • header-{name}.php
    • footer-{name}.php
    • sidebar-{name}.php
    • {slug}-{name}.php
    • searchform.php
    • comment.php
  2. get_template_part( string $slug, string $name = null ) :: wp:f:get_template_part

    • require file {slug.php} or {slug}-{name}.php 

      <?php get_template_part( 'template-parts/header/header', 'image' ); ?>
// require( 'path-to-theme/template-parts/header/header-image.php' );
    • Call locate_template
      • Call load_template
    • wp:action:get_template_part$slug

    • Pass variables to template 

      // parent
$args = ['...'];
set_query_var( 'my_query_args', $args );

get_template_part('template-parts/test');

// test.php
$query = new WP_Query( $my_query_args );
    get_bloginfo( $show = '', $filter = 'raw' )
    retrieves info about the current site
    bloginfo( $show ='' )
    echo get_bloginfo( $show, 'display' )
  3. wp_editor wp:f:wp_editor
    Render an editor for field in a page in the typical fashion used in Posts and Pages
    Doc
    (no term)
    May be used in
    (no term)
    Use wp:f:update_post_meta in wp:action:save_post for storing the new field
  4. has_custom_logo, get_custom_logo, the_custom_logo wp:custom logo:display

    get_custom_logo returns markup the_custom_logo displays markup 

    $custom_logo_id = get_theme_mod( 'custom_logo' );
$logo = wp_get_attachment_image_src( $custom_logo_id , 'full' );
if ( has_custom_logo() ) {
    echo '<img src="'. esc_url( $logo[0] ) .'">';
} else {
    echo '<h1>'. get_bloginfo( 'name' ) .'</h1>';
}
3.16.9.5. author-template.php
  1. get_the_author_meta( string $field = '', int|false $user_id = false ) : string wp:f:get_the_author_meta
    $field
    can be
    • display_name
    $user_id
    default false to get the current post's user. Can be $post->post_author
3.16.9.6. bookmark_template.php
3.16.9.7. comment-template.php
wp_list_comments( $args = array(), $comments = null)
wp:f:wp_list_comments
  • List comments in comments.php template. Use wp:filter:wp_list_comments_args
comment_form( $args = array(), $post_id = null)
outputs comment form wp:f:comment_form
comments_template( $file = '/comments.php', $separate_comments = false )
wp:f:comments_template
  • Default loads wp-includes/theme-compat/comments.php
3.16.9.8. link-template.php
get_permalink( int|WP_Post $post = 0, bool $leavename = false)
get full permalink for current post or post ID
$post
optional, default global $post
$leavename
whether to keep post/page name
return string|false
(no term)
Applied filters
the_permalink( int|WP_Post $post = 0 )
print escaped get_permalink
Applied filters
wp:filters:the_permalink
edit_post_link( $text = null, $before = '', $after = '', $id = 0, $class = 'post-edit-link' )
print edit post link
  • edit_post_link( __( '<span class="small">(Edit)</span>', 'genesis' ) );
get_theme_file_uri( $file = '' )
wp:f:get_theme_file_uri

  • Check for file existance and returns URL. Get from child theme first js/my-scripts.js. Fall back to parent theme 

    wp_enqueue_script( 'my-script', get_theme_file_uri( 'js/my-script.js' ) );
get_theme_file_path( $file = '' )
wp:f:get_theme_file_path

  • Return the path in filesystem so that you can use filemtime 

    wp_enqueue_script(
'my-script',
get_theme_file_uri( 'js/my-script.js' ),
array(),
filemtime( get_theme_file_path( 'js/my-script.js' ) )
);
3.16.9.9. media-template.php
3.16.9.10. nav-menu-template.php
3.16.9.11. feed.php
the_title_rss()
echo get_the_title_rss() wp:f:the_title_rss
(no term)
get_the_title_rss()

3.16.10. Post API wp-includes/post.php wp:api:post

3.16.10.1. get_post( int|WP_Post|null $post = null, string $output = OBJECT, string $filter = 'raw' ) wp:f:get_post
  • Default is global $post
  • Return wp:post object
3.16.10.2. get_metadata( string $meta_type, int $object_id, string $meta_key = '', bool $single = false ) wp:f:get_metadata
$meta_key
default to return data for all keys
$single
wheter to return single value. return array if it's false
(no term)

e.g. Return post thumbnail alt text 

// return post thumbnail alt text
$thumb_id = get_post_thumbnail_id(get_the_ID());
$alt = get_post_meta($thumb_id, '_wp_attachment_image_alt', true);
if( $alt ):
echo $alt;
endif;

// return a post metadata
$meta = get_post_meta(get_the_ID());
echo $meta['header-title'][0];
3.16.10.3. get_post_meta( int $post_id, string $key = '', bool $single = false ) wp:f:get_post_meta
return get_metadata( 'post', $post_id, $key, $single );
refer to wp:f:get_metadata
3.16.10.4. get_posts( $args = null ) : WP_Post[]|int[] wp:f:get_posts
// new WP_Query(); is created
$args = array(
    'post_type'      => 'speaker',
    'posts_per_page' => -1,
);
$r = get_posts($args);

// run loop

// reset global $post
wp_reset_postdata();
3.16.10.5. wp_get_post_terms( $post_id = 0, $taxonomy = 'post_tag', $args = array() ) : WP_Term[]|WP_Error wp:f:wp_get_post_terms
  • Retrieve terms for a post
    $taxonomy
    default only return terms that are tags
    $post_id
    default current post if it's in a loop
    $args
    default $args = array('orderby' => 'name', 'order' => 'ASC', 'fields' => 'all');
    Return
    an array of taxonomy terms as objects, empty array or WP_Error
  • wp_get_post_terms( $post_id, 'publication', ['field' => 'slugs'] )

  • Sample 

    Array
(
[0] => WP_Term Object
    (
    [term_id] => 145
    [name] => Example Category
    [slug] => example-cat
    [term_group] => 0
    [term_taxonomy_id] => 145
    [taxonomy] => adcpt_categories
    [description] => 
    [parent] => 0
    [count] => 2
    [filter] => raw
    )

)

    Return all terms for a post 

    global $post;
$tax = get_taxonomies();
$terms = wp_get_post_terms($post->ID, $tax);
3.16.10.6. add_post_type_support( string $post_type, string|array $supports) wp:f:add_post_type_support
  • remove_post_type_support( string $post_type, string $supports )
  • post_type_supports( string $post_type, string $feature )
  • string

  • string/array

    • title
    • content
    • author
    • current theme must support Post Thumbnails
    • excerpt
    • trackbacks
    • wp:Custom_Fields. Don't have to add 'custom-fields' in order to use wp:plugin:acf
    • comments
    • revisions
    • menu order, hierarchical must be true
    • add post formats
    add_action('init', 'lili_custom_post_type');
function lili_custom_post_type() {
  $labels = [
    // array, wp:register_post_type:labels
  ];

  $args = [
    'label' => 'Plural Name', // string. required.
    'labels' => $labels,
    'description' => '', // string, optional
    'public' => false,
    /* Default. bool.
 *       exclude_from_search, publicly_queryable, show_in_nav_menus, show_ui
 * true:               false,               true,              true, true
 * false:               true,              false,             false, false
 */
    'exclude_from_search' => false, // default. optional. 'site/?s=search-term'
    'publicly_queryable' => true, // optional. take 'public' value.
    /* ?post_type={post_type_key},
 * ?{post_type_key}={single_post_slug},
 * ?{post_type_query_var}={single_post_slug}
 */
    'show_ui' => true, // optional. bool. default take 'public' value.
    'show_in_nav_menus' => true, // optional. bool. inherits 'public' value.
    'show_in_menu' => true, // optional, bool or string. default inherits show_ui
    // false :: do not display in admin menu
    // true :: display as a top level menu
    // 'some string' :: display as a submenu of a menu 'some string'
    'show_in_admin_bar' => true, // optional, bool, default inherits show_in_
    'supports' => ['title', 'editor'], // optional. array/bool. wp:f:add_post_type_support

  ];

  register_post_type('movies', $args );

}

    Use wp:action:pre_get_posts to enable query for this new post type

3.16.10.7. register_post_type wp:f:register_post_type wp:add cpt
add_action('init', 'lili_custom_post_type');
function lili_custom_post_type() {
  $labels = [
    // array, wp:register_post_type:labels
  ];

  $args = [
    'label' => 'Plural Name', // string, optional, default labels['name']
    'labels' => $labels,
    'description' => '', // string, optional

    'public' => false,
    /* Default false
 *       exclude_from_search, publicly_queryable, show_in_nav_menus, show_ui
 * true:               false,               true,              true, true
 * false:               true,              false,             false, false
 */

    'exclude_from_search' => false, // default. optional. 'site/?s=search-term'

    'publicly_queryable' => true, // optional. take 'public' value.
    /* ?post_type={post_type_key},
 * ?{post_type_key}={single_post_slug},
 * ?{post_type_query_var}={single_post_slug}
 */

    // 'query_var' => true, // bool|string d:true to use $post_type. Set to false means the post type cannot be loaded at /?{query_var}={single_post_slug}. It has no effect when public_queryable is false

    'show_ui' => true, // optional. bool. default take 'public' value.
    'show_in_nav_menus' => true, // optional. bool. inherits 'public' value.
    'show_in_menu' => true, // optional, bool or string. default inherits show_ui
    // false :: do not display in admin menu
    // true :: display as a top level menu
    // 'some string' :: display as a submenu of a menu 'some string'
    'show_in_admin_bar' => true, // optional, bool, default inherits show_in_menu

    'capability_type' => 'post', // default, for naming capabilities e.g. `edit_published_posts`
    'map_meta_cap' => false, // default false. If you specify capability_type other than post, you should set this to true. Optionally, build map_meta_cap hooks to give further granular meta capabilities

    'supports' => ['title', 'editor'], 
    // optional. array/bool. wp:f:add_post_type_support
    // 'custom-fields' :: don't have to add in order to use wp:plugin:acf
    // 'title', 'editor' (content) this is the default
    // 'author','excerpt', 'trackbacks',
    // 'thumbnail' (featured image, current theme must also support post-thumbnails)
    // 'comments' (also will see comment count balloon on edit screen)
    // 'revisions' (will store revisions)
    // 'page-attributes' (menu order, hierarchical must be true to show Parent option)
    // 'post-formats' add post formats, see Post Formats

    // 'hierarchical' => true
    // default false, when true 'supports' should contain 'page-attributes'  

    'has_archive' => false, // d:false

        'rewrite' => [
            'slug' => 'movies', // d:$post_type
            'with_front' => true, // d:true should the permalink strucutre to be prepended with the front base
            'feeds' => false, // d:$has_archive should a feed permalink structure be built
            'pages' => true, // d:true should the permalink structure provide for pagination
            'ep_mask' => EP_PERMALINK, // don't know what it is..
        ], // bool|array d:true and use $post_type as slug

  ];

  register_post_type('movies', $args);

}

add_action( 'pre_get_posts', 'lili_pre_get_posts' );
// Should not call this in template because in template the $query is already run
function lili_pre_get_posts( $query ) {
  if ( ! is_page() && ! is_home() && $query->is_main_query() ) {
    $query->set( 'post_type', array( 'post', 'movies' ) );
  }
  // this function should return nothing.
}

Get Custom Post Types 

$cpts = get_post_types( array(
    'public'   => true,
    '_builtin' => false,
));
  1. labels wp:register_post_type:labels
    'labels' => array(
  'name' => __( 'Jobs' ),
  'singular_name' => __( 'Job' )
)
  2. menu_icon

    optional, default null > posts icon 'dashicons-video-alt' (Uses the video icon from Dashicons) 'get_template_directory_uri() . "/images/cutom-posttype-icon.png"' (Use a image located in the current theme) 'data:image/svg+xml;base64,' . base64_encode( "<svg version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px" width="20px" height="20px" viewBox="0 0 459 459"> <path fill="black" d="POINTS"/></svg>" )' (directly embedding a svg with 'fill="black"' will allow correct colors.)

3.16.10.8. register_post_status wp:f:register_post_status
  • Register post status
  • Do not use before wp:action:init
  • add to wp:global:wp_post_statuses
  • Built-in post statuses _builtin
    publish
    public
    future
    protected
    draft
    protected
    pending
    protected
    private
    private
    trash
    internal
    auto-draft
    internal
    inherit
    internal Used with child post such as Attachments (e.g. featured image) and Revisions
    request-pending
    internal
    request-confirmed
    internal
    request-failed
    internal
    request-completed
    internal
  • Extra post statuses
    wp:plugin:acf
    acf-disabled
  • register_post_status( string $post_status, array|string $args = array() )

    • $args 

      $args = [
    'label' => 'Post status name for translation', // bool|string d:$post_status
    'label_count' => '', // bool|array d:none text to display on admin screen e.g. _n_noop( 'Unread <span class="count">(%s)</span>', 'Unread <span class="count">(%s)</span>' )
    'exclude_from_search' => false, // bool d:$internal whether to exclude posts with this post status
    'internal' => false, // bool d:false whether the status is for internal use only
    'public' => false, // bool whether posts of this status should be shown in the front end of the site
    // '_builtin' => false, // bool d:false core-use only
    'protected' => false, // bool d:false whether posts with this status should be protected
    'private' => false, // bool d:false whether posts with this status should be private
    'publicly_queryable' => false, // bool d:$public whether posts with this status should be publicly-queryable
    'show_in_admin_all_list' => false, // bool d:$internal whether to include posts in the edit listing for their post type
    'show_in_admin_status_list'=> false, // bool d:$internal show in the list of statuses with post counts at the top of the edit listings e.g. All (12) | Published (9) | My Custom Status (2)
];

function my_custom_post_status(){
    register_post_status( 'unread', array(
        'label'                     => _x( 'Unread', 'post' ),
        'public'                    => true,
        'exclude_from_search'       => false,
        'show_in_admin_all_list'    => true,
        'show_in_admin_status_list' => true,
        'label_count'               => _n_noop( 'Unread <span class="count">(%s)</span>', 'Unread <span class="count">(%s)</span>' ),
    ) );
}
add_action( 'init', 'my_custom_post_status' );
3.16.10.9. register_post_meta wp:f:register_post_meta
  • register_post_meta( $post_type, $meta_key, array $args ) : bool
    • register_meta( 'post', $meta_key, $args )
3.16.10.10. update_post_meta wp:f:update_post_meta

3.16.11. Post Formats wp-includes/post-formats.php

3.16.11.1. has_post_format wp:f:has_post_format

In template 

if ( has_post_format( 'aside' )) {
  // code to display the aside format post here
} else if (has_post_format('gallery')) {
    // stuff to display the gallery format post here
} else if (has_post_format('link')) {
    // stuff to display the link format post here
}else {
    // code to display the normal format post here
}

3.16.12. Rewrite API wp:api:rewrite

3.16.12.1. add_rewrite_rule, add_rewrite_tag
  • wp:f:add_rewrite_rule
    Need to flush
    Settings > Permalinks > click Save Changes without any changes to add into wp:options:rewrite_rules
    (no term)
    Add a rewrite rule that transforms a URL structure to a set of query vars
    (no term)
    Use with wp:f:add_rewrite_tag
    (no term)
    wp:action:init
    (no term)
    add_rewrite_rule( string $regex, string|array $query, string $after = 'bottom' )
    $query
    $matches[] to retrieve the values of a matched URL, capture group data starts at 1, not 0
    $after
    This can either be 'top' or 'bottom'. 'top' will take precedence over WordPress's existing rules, where 'bottom' will check all other rules match first
    (no term)

    Sample 

    function custom_rewrite_basic() {
  add_rewrite_rule('^leaf/([0-9]+)/?', 'index.php?page_id=$matches[1]', 'top');
}
add_action('init', 'custom_rewrite_basic');
  • wp:f:add_rewrite_tag
    • add a query var wp:filter:query_vars

    • Sample 

      function custom_rewrite_tag() {
  add_rewrite_tag('%food%', '([^&]+)');
  add_rewrite_tag('%variety%', '([^&]+)');
}
add_action('init', 'custom_rewrite_tag', 10, 0);

function custom_rewrite_rule() {
  add_rewrite_rule('^nutrition/([^/]*)/([^/]*)/?','index.php?page_id=12&food=$matches[1]&variety=$matches[2]','top');
}
add_action('init', 'custom_rewrite_rule', 10, 0);

Create a page called Nutrition with page id 12 and use a custom template my-custom-template.php 

/**
 * Template Name: Nutritional Information
 */
get_header(); 

global $wp_query;
echo 'Food : ' . $wp_query->query_vars['food'];
echo '<br />';
echo 'Variety : ' . $wp_query->query_vars['variety'];
// ... more ...
get_footer();
3.16.12.2. add_feed( string $feedname, callback $function ) : string wp:f:add_feed wp:rss wp:feed
Default template
/wp-includes/feed-rss2.php
Functions to use in The Loop (template)
wp:api:feed
add action
wp:action:do_feed_$feedname which does the callback $function at p:10
return the action name
do_feed_$feedname
(no term)
Refer to wp:f:do_feed
function lili_RSS() {
    add_feed('lili-editorial', 'lili_editorial_RSS');
}
add_action('init', 'lili_RSS');

function lili_editorial_RSS() {
    get_template_part('rss','feedname');
    // rss is slug and feedname is an extra name
    // Search in order
    // /themes/child/rss-feedname.php
    // /themes/parent/rss-feedname.php
    // /themes/child/rss.php
    // /themes/parent/rss.php

    // the feed url is http://a.ca/feed/lili-editorial which is the feed name defined in add_feed
}

3.16.13. Feed API wp-includes/feed.php wp:api:feed

<?php
$args = array(
    'post_type' => array('blog'), // e.g. any
    'tag' => 'editors-choice',
    'orderby' => 'date', // Default. e.g. title
    'order' => 'DESC', // Default. e.g. ASC
);
?>
query_posts($args);
<?php while (have_posts()) : the_post(); ?>
<item>
  <title><?php the_title_rss(); ?></title>
  <?php if (has_post_thumbnail($post->ID)) : ?>
  <?php
    $image = wp_get_attachment_image_src(
    get_post_thumbnail_id( $post->ID ),
    'thumbnail'
    );
  ?>
  <feature_image><?php echo $image[0]; ?></feature_image>
  <?php endif; ?>
  <pubDate><?php echo mysql2date('D, d M Y H:i:s +0000',
                get_post_time('Y-m-d H:i:s', true),
                false); ?></pubDate>
  <?php // Text Version ?>
  <excerpt><![CDATA[<<?php echo get_the_exerpt(); ?>]]></excerpt>
  <excerpt_html><![CDATA[<<?php echo the_exerpt_rss(); ?>]]></excerpt_html>
  <?php echo the_category_rss('rss2'); // categories and tags ?>
</item>  
<?php endwhile; ?>

3.16.14. Dependency API wp-includes/script-loader.php wp:api:dependency

3.16.14.1. wp_localize_script wp:f:wp_localize_script
  • Must be called after the script has been registered using wp_register_script() or wp_enqueue_script()
  • Use to define variables as an object
// register a script that may be loaded using wp_enqueue_script()
// wp_register_script( 'unique_handle', 'path/to/myscript.js' );
// later
// wp_enqueue_script( 'unique_handle' );

// or you enque directly wp:f:wp_enqueue_script
wp_enqueue_script( 'unique_handle', get_theme_file_uri( '/path/to/myscript.js' ) );

// Pass vars in $translation_array to file path/to/myscript.js (registered using handle `unique_handle`)
$translation_array = array(
  'some_string' => 'hello';
  'a_value' => '10'
);
wp_localize_script( 'unique_handle', 'object_name', $translation_array );
wp_enqueue_script( 'unique_handle' );

In the enqueued javascript file path/to/myscript.js 

console.log( object_name.some_string ); // default all values are strings
parseInt( object_name.some_int, 10 ); // pass parameter as int

If you just want to define pass PHP variables and don't want to create empty js files 

wp_localize_script( 'jquery', 'mynamesapce', array('ajaxurl' => admin_url( 'admin-ajax.php' ) ) );

wp_enqueue_script( 'twentyseventeen-skip-link-focus-fix', get_theme_file_uri( '/assets/js/skip-link-focus-fix.js' ), array(), '1.0', true );
wp_localize_script( 'twentyseventeen-skip-link-focus-fix', 'twentyseventeenScreenReaderText', $twentyseventeen_l10n );
3.16.14.2. wp_enqueue_script, wp_script_add_data wp:f:wp_enqueue_script
wp_enqueue_script( string $handle, string $src = '', array $deps = array(), string|bool|null $ver = false, bool $in_footer = false )

wp_enqueue_script( 'html5', get_theme_file_uri( '/assets/js/html5.js' ), array(), '3.7.3' );
wp_script_add_data( 'html5', 'conditional', 'lt IE 9' );

When a script is added with dependency on jQuery, jQuery is auto loaded by wp. Change jQuery 

wp_deregister_script('jquery');
wp_enqueue_script('jquery', 'https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js', array(), null, true);
wp_enqueue_script('my-custom-script', get_template_directory_uri() .'/js/my-custom-script.js', array('jquery'), null, true);

If there's script depends on jquery and it's loaded in header, even though jquery is set to load in footer, jquery will be loaded in header.

3.16.14.3. wp_enqueue_style, wp_style_add_data wp:f:wp_enqueue_style wp:f:wp_style_add_data
  • Used in wp:action:wp_enqueue_scripts
  • Use wp:filter:script_loader_src
  • wp_enqueue_style( string $handle, string $src = '', array $deps = array(), string|bool|null $ver = false, string $media = 'all' )
  • (array) (Optional) An array of registered stylesheet handles this stylesheet depends on
    • Default value: array()
  • (string|bool|null) (Optional) String specifying stylesheet version number, if it has one, which is added to the URL as a query string for cache busting purposes. If version is set to false, a version number is automatically added equal to current installed WordPress version. If set to null, no version is added
    • Default value: false
  • (string) (Optional) The media for which this stylesheet has been defined. Accepts media types like 'all', 'print' and 'screen', or media queries like '(orientation: portrait)' and '(max-width: 640px)'
    • Default value: 'all'
function wpb_add_google_fonts() {
    wp_enqueue_style( 'wpb-google-fonts', 'http://fonts.googleapis.com/css?family=Open+Sans:300italic,400italic,700italic,400,700,300', false ); 
}

add_action( 'wp_enqueue_scripts', 'wpb_add_google_fonts' );

Conditional statements are removed from IE 10 or IE 11. Refer to css:font-face 

// Load the main stylesheet
    wp_enqueue_style( 'my-theme', get_stylesheet_uri() );

/**
 * Load our IE-only stylesheet for all versions of IE:
 * <!--[if IE]> ... <![endif]-->
 */
wp_enqueue_style( 'my-theme-ie', get_stylesheet_directory_uri() . "/css/ie.css", array( 'my-theme' ) );
wp_style_add_data( 'my-theme-ie', 'conditional', 'IE' );

/**
 * Load our IE version-specific stylesheet:
 * <!--[if IE 7]> ... <![endif]-->
 */
wp_enqueue_style( 'my-theme-ie7', get_stylesheet_directory_uri() . "/css/ie7.css", array( 'my-theme' ) );
wp_style_add_data( 'my-theme-ie7', 'conditional', 'IE 7' );

/**
 * Load our IE specific stylesheet for a range of older versions:
 * <!--[if lt IE 9]> ... <![endif]-->
 * <!--[if lte IE 8]> ... <![endif]-->
 * NOTE: You can use the 'less than' or the 'less than or equal to' syntax here interchangeably.
 */
wp_enqueue_style( 'my-theme-old-ie', get_stylesheet_directory_uri() . "/css/old-ie.css", array( 'my-theme' ) );
wp_style_add_data( 'my-theme-old-ie', 'conditional', 'lt IE 9' );

/**
 * Load our IE specific stylesheet for a range of newer versions:
 * <!--[if gt IE 8]> ... <![endif]-->
 * <!--[if gte IE 9]> ... <![endif]-->
 * NOTE: You can use the 'greater than' or the 'greater than or equal to' syntax here interchangeably.
 */
wp_enqueue_style( 'my-theme-new-ie', get_stylesheet_directory_uri() . "/css/new-ie.css", array( 'my-theme' ) );
wp_style_add_data( 'my-theme-ie', 'conditional', 'gt IE 8' );

3.16.15. Taxonomy API wp-includes/taxonomy.php wp:api:taxonomy

3.16.15.1. get_term_by( $field, $value, $taxonomy = '', $output = OBJECT, $filter = 'raw' )
  • Return WP_Term|array|false
  • get_term_by( 'slug', 'my-term-slug', 'taxonomy-of-the-term' )
  • get_term_by( 'slug', get_query_var( 'term' ), get_query_var( 'taxonomy' ) )
  • slug, name, id or term_taxonomy_id
  • OBJECT, ARRAY_A, ARRAY_N
3.16.15.2. register_taxonomy( string $taxonomy, array|string $object_type, array|string $args = array() ) wp:f:register_taxonomy
add_action( 'init', 'create_topics_hierarchical_taxonomy', 0 );

//create a custom taxonomy name it topics for your posts

function create_topics_hierarchical_taxonomy() {

// Add new taxonomy, make it hierarchical like categories
//first do the translations part for GUI

  $labels = array(
    'name' => _x( 'Topics', 'taxonomy general name' ),
    'singular_name' => _x( 'Topic', 'taxonomy singular name' ),
    'search_items' =>  __( 'Search Topics' ),
    'all_items' => __( 'All Topics' ),
    'parent_item' => __( 'Parent Topic' ),
    'parent_item_colon' => __( 'Parent Topic:' ),
    'edit_item' => __( 'Edit Topic' ), 
    'update_item' => __( 'Update Topic' ),
    'add_new_item' => __( 'Add New Topic' ),
    'new_item_name' => __( 'New Topic Name' ),
    'menu_name' => __( 'Topics' ),
  );    

// Now register the taxonomy

  register_taxonomy('topics',array('post'), array(
    'hierarchical' => true,
    'labels' => $labels,
    'show_ui' => true,
    'show_admin_column' => true,
    'query_var' => true,
    'rewrite' => array( 'slug' => 'topic' ),
  ));

}

Non-hierarchical taxonomy 

add_action( 'init', 'create_topics_nonhierarchical_taxonomy', 0 );

function create_topics_nonhierarchical_taxonomy() {

// Labels part for the GUI

  $labels = array(
    'name' => _x( 'Topics', 'taxonomy general name' ),
    'singular_name' => _x( 'Topic', 'taxonomy singular name' ),
    'search_items' =>  __( 'Search Topics' ),
    'popular_items' => __( 'Popular Topics' ),
    'all_items' => __( 'All Topics' ),
    'parent_item' => null,
    'parent_item_colon' => null,
    'edit_item' => __( 'Edit Topic' ), 
    'update_item' => __( 'Update Topic' ),
    'add_new_item' => __( 'Add New Topic' ),
    'new_item_name' => __( 'New Topic Name' ),
    'separate_items_with_commas' => __( 'Separate topics with commas' ),
    'add_or_remove_items' => __( 'Add or remove topics' ),
    'choose_from_most_used' => __( 'Choose from the most used topics' ),
    'menu_name' => __( 'Topics' ),
  ); 

// Now register the non-hierarchical taxonomy like tag

  register_taxonomy('topics','post',array(
    'hierarchical' => false,
    'labels' => $labels,
    'show_ui' => true,
    'show_admin_column' => true,
    'update_count_callback' => '_update_post_term_count',
    'query_var' => true,
    'rewrite' => array( 'slug' => 'topic' ),
  ));
}
3.16.15.3. wp_update_term_count_now( array $terms, string $taxonomy ) wp:f:wp_update_term_count_now
$terms
array of term_taxonomy_id
return true
when complete
(no term)
It's only called when
  • a post's post status is updated (update count for the post's associated terms)
  • a tax is added or removed (update count for that tax only)
  • Not cleared when cache is cleared and has to be run manually to update count
call_user_func( $taxonomy->update_count_callback, $terms, $taxonomy )
if wp:f:register_taxonomy:update_count_callback is defined
_update_post_term_count( $terms, $taxonomy )
if there's any attached post types that exist, get total count of publish posts of attached post types
_update_generic_term_count( $terms, $taxonomy )
if there's no attached post type that exists, get total count of anything from wp:db:wp_term_relationships
3.16.15.4. wp:action:registered_taxonomy
3.16.15.5. Add a sortable column to taxonomy edit page
Taxonomy
sponsor_type
(no term)
Refer to wp:api:plugin for adding a column to custom post type
Filter manage_${taxonomy}_custom_column
wp:api:plugin:filter:managetaxonomy_custom_column
<?php
add_filter( 'manage_edit-sponsor_type_columns', function ( $columns ) {
    $columns['my_term_id']             = __( 'Term ID' ); // add a column
    $columns['sponsorship_type_order'] = __( 'Order' ); // add a column
    //
    // to see existing columns, wp-admin/edit-tags.php?taxonomy=sponsor_type
    //var_dump('hello'); var_dump($columns);
    //unset( $columns['cpt-shows'] );                 // Don't allow sort by a column
    return $columns;
} );

// populate values in column Order
add_filter( 'manage_sponsor_type_custom_column', function ( $value, $column_name, $term_id ) {
    $term  = get_term( $term_id, 'sponsor_type' );
    $order = get_field( 'sponsorship_type_order', $term );
    switch ( $column_name ) {
        case 'sponsorship_type_order':
            $value = $order;
            break;
        default:
            break;
    }

    return $value;
}, 10, 3 );

// make the column sortable
add_filter( 'manage_edit-sponsor_type_sortable_columns', function ( $columns ) {
    // see existing sortable columns
    //var_dump('hello'); var_dump($columns);

    $columns['sponsorship_type_order'] = 'custom_order'; // don't use order or orderby as the url parameter name

    return $columns;
} );

// add sort mechanism
add_filter( 'terms_clauses', function ( $pieces, $taxonomies, $args ) {
    global $pagenow;
    if ( ! is_admin() ) {
        return $pieces;
    }
    if ( is_admin() && $pagenow == 'edit-tags.php' && $taxonomies[0] == 'sponsor_type' &&
     ( ! isset( $_GET['orderby'] ) || $_GET['orderby'] == 'custom_order' )
    ) {
        // to further customize the sorting. By default, using the populated value to sort is good enough
        // e.g. If orderby is not set, add some default from wp_options
        // $pieces['join']   .= " INNER JOIN wp_options AS opt ON opt.option_name = concat('issue_',t.term_id,'_issue_date')";
        // $pieces['orderby'] = "ORDER BY opt.option_value";
        // $pieces['order']   = isset($_GET['order']) ? $_GET['order'] : "DESC";
    }

    return $pieces;
}, 10, 3 );
3.16.15.6. get_taxonomies( $args = array(), $output = 'names', $operator = 'and' ) : string[]|WP_Taxonomy[] wp:f:get_taxonomies
// @return string[]|WP_Taxonomy[] An array of taxonomy names or objects.
$return = [
    'category' => 'category',
    'post_tag' => 'post_tag',
    'custom_taxonomy_name' => 'custom_taxonomy_name',
    // ...
];

3.16.16. Shortcode API wp-includes/shortcodes.php wp:api:shortcode

3.16.16.1. add_shortcode wp:f:add_shortcode
// [bar-tag foo="foo-value"]

// Use underscore instead of hyphen e.g. bar_tag
function bartag_func( $atts, $content, $tag ) {
  $a = shortcode_atts( array(
'class' => 'something', // default value
'attr_2' => 'something else', // 2nd attribute
  ), $atts );

  // get current post
  // global $post; $post->ID

  // Use return not echo
  return '<span class="'.esc_attr($a['class']).'">'
     .$content
     // or
     // .do_shortcode($content)
     ."foo = {$a['foo']}"
     .'</span>';

/*
ob_start();
?> <HTML> <here> ... <?php
return ob_get_clean();
*/

/*
ob_start();
get_template_part( 'template-parts/form/contact-us');
return ob_get_clean();
*/

}
add_shortcode( 'bar-tag', 'bartag_func' );

// In order to run shortcode in Widgets, add this
add_filter('widget_text','do_shortcode');
// wp:filter:widget_text

Pass array to shortcode attribute 

$data = [
[],[],[]
];

var_dump(serialize($data));

// use single quotes
// [your-shortcode data='serialized-string']

function your_shortcode_cb() {
// ...
$data = unserialize($a['data']);
}
3.16.16.2. shortcode_unautop wp:f:shortcode_unautop
add_filter( 'the_content', 'shortcode_unautop'  );

3.16.17. OEmbed API wp-includes/embed.php wp:api:oembed

3.16.17.1. Basics
  • Shortcode [embed width="560" height="315"]https://youtu.be/abc[/embed]
    Whitelist of sources
    https://wordpress.org/support/article/embeds/
  • Embeds that aren't associated with a post will be cached in oembed_cache post type
  • Embeds that are associated with a post is stored in the post's metadata

3.16.18. Media API wp-includes/media.php wp:api:media

3.16.18.1. add_image_size wp:f:add_image_size
  • Use wp:action:after_setup_theme
  • add_image_size( string $name, int $width, int $height, bool|array $crop = false )
  • Every newly uploaded image will have image files with all defined image sizes
  • $crop can be
    false
    will be scaled
    true
    will be cropped to the specified dimensions using center positions
    array( $x_crop_position, $y_crop_position )
    will be cropped to the specified dimensions within the defined crop area
    $x_crop_position
    'left', 'center' or 'right'
    $y_crop_position
    'top', 'center' or 'bottom'
  • If the original image (e.g. featured image) is smaller than the thumbnail image size, by default, neither it's set to crop to scale, the result thumbnail will take the original image without scaling up

  • Refer to wp:filter:image_size_names_choose 

    function lili_theme_setup() {
 add_image_size('lili-w700', 700, 9999); // fixed width, unlimited height
 add_image_size('lili-w700h600', 700, 600, true); // crop in exact dimension 
}

    Force to scale up the thumbnail if original image is smaller than the thumbnail size, and maintain ratio

    • Does not seem to work for scale (non-croppable) image sizes last time I tried.. Ratio is not maintained for scaling image sizes
    function alx_thumbnail_upscale( $default, $orig_w, $orig_h, $new_w, $new_h, $crop ){
  // comment the line below to enable upscale for both scale and crop
    if ( !$crop ) return null; // let the wordpress default function handle automatic scale

    $aspect_ratio = $orig_w / $orig_h;
    $size_ratio = max($new_w / $orig_w, $new_h / $orig_h);

    $crop_w = round($new_w / $size_ratio);
    $crop_h = round($new_h / $size_ratio);

    $s_x = floor( ($orig_w - $crop_w) / 2 );
    $s_y = floor( ($orig_h - $crop_h) / 2 );

    return array( 0, 0, (int) $s_x, (int) $s_y, (int) $new_w, (int) $new_h, (int) $crop_w, (int) $crop_h );
}
add_filter( 'image_resize_dimensions', 'alx_thumbnail_upscale', 10, 6 );

    // after this call, my-image-size-style-name thumbnail should be ready
// by default, wp does not scale up if the original file is smaller than the requested thumbnail image size
// regardless of the add_image_size setting
// Original my-image-size-style-name setting: add_image_size( 'my-image-size-style-name', 340, 200, true );
regenerateImageSizeIfOneIsMissing( get_post_thumbnail_id( $post->ID ), 'my-image-size-style-name' );
$image = wp_get_attachment_image_src( get_post_thumbnail_id( $post->ID ), 'my-image-size-style-name' );

// regenerate all image sizes for an attachment (e.g. featured image) if the specified image size does not exist
function regenerateImageSizeIfOneIsMissing( $attachment_id, $check_image_size, $enforce_enlarge = false ) {
    global $_wp_additional_image_sizes;

    // stop if no file id is provided or the image size is not defined
    if ( ! $attachment_id || ! isset( $_wp_additional_image_sizes[ $check_image_size ] ) ) {
        return false;
    }
    //print_r($attachment_id);
    //print '<pre>';    print_r( $_wp_additional_image_sizes ); print '</pre>';

    $metadata = wp_get_attachment_metadata( $attachment_id );
    if ( ! ( isset( $metadata['sizes'] ) && isset( $metadata['sizes'][ $check_image_size ] ) ) ) {
        //printf('Image size "%s" does not exist, regenerate all image sizes for this attachment', $check_image_size);

        $fullsizepath = get_attached_file( $attachment_id );

        if ( false === $fullsizepath || ! file_exists( $fullsizepath ) ) {
            // stop if there is no full file or it does not exist
            return false;
        }

        if ( ! function_exists( 'wp_generate_attachment_metadata' ) ) {
            require_once( ABSPATH . 'wp-admin/includes/image.php' );
        }

        $r = false;

        // By default, wp does not scale up if the original file is smaller than the requested thumbnail image size
        // regardless of the add_image_size setting
        if ($enforce_enlarge) {
            add_filter( 'image_resize_dimensions', 'alx_thumbnail_upscale', 10, 6 );
        }

        // regenerate all image sizes and metadata
        $regenerated_metadata = wp_generate_attachment_metadata( $attachment_id, $fullsizepath );

        // merge original metadata
        $regenerated_metadata['sizes'] = array_merge( $metadata['sizes'], $regenerated_metadata['sizes'] );
        //echo 'original metadata'; print_r($metadata);
        //echo 'regenerated metadata'; print_r($regenerated_metadata);

        // update metadata
        if ( wp_update_attachment_metadata( $attachment_id, $regenerated_metadata ) ) {
            $r = true;
            //print 'regenerate metadata result successful';
        } else {
            //print 'regenerate metadata result not successful';
        }

        // remove filter so that wp is back to default for thumbnail resizing
        if ($enforce_enlarge) {
            remove_filter( 'image_resize_dimensions', 'alx_thumbnail_upscale', 10, 6 );
        }

        return $r;

    } else {
        //printf('Image size "%s" exists, regeneration is not needed', $check_image_size);
        //  print_r($metadata['sizes'][$check_image_size]);
    }
}
3.16.18.2. wp_get_image_editor( string $path, array $args = array() ) : WP_Image_Editor|WP_Error

  • Returns WP_Error or WP_Image_Editor 

    function image_resize( $file, $max_w, $max_h, $crop = false, $suffix = null, $dest_path = null, $jpeg_quality = 90 ) {
_deprecated_function( __FUNCTION__, '3.5.0', 'wp_get_image_editor()' );

$editor = wp_get_image_editor( $file );
if ( is_wp_error( $editor ) )
    return $editor;
$editor->set_quality( $jpeg_quality );

$resized = $editor->resize( $max_w, $max_h, $crop );
if ( is_wp_error( $resized ) )
    return $resized;

$dest_file = $editor->generate_filename( $suffix, $dest_path );
$saved = $editor->save( $dest_file );

if ( is_wp_error( $saved ) )
    return $saved;

return $dest_file;
}

3.16.19. HTTP Request API wp-includes/httpl.php wp:api:http request

3.16.19.1. wp_remote_get, wp_remote_post, wp_remote_head, wp_remote_request
$response = wp_remote_get( 'http://www.example.com/index.html' );
// url has to include protocal e.g. http://
if ( is_array( $response ) ) {
  $header = $response['headers']; // array of http header lines
  $body = $response['body']; // use the content
}

wp_remote_get( 'http://www.example.com/index.php?action=foo', array( 'timeout' => 120, 'httpversion' => '1.1' ) );

Methods to work with $response

  • wp_remote_retrieve_body() - Retrieves just the body from the response.
  • wp_remote_retrieve_header() - Gives you a single HTTP header based on name from the response.
  • wp_remote_retrieve_headers() - Returns all of the HTTP headers in an array for processing.
  • wp_remote_retrieve_response_code() - Gives you the number for the HTTP response. This should be 200, but could be 4xx or even 3xx on failure.
  • wp_remote_retrieve_response_message() - Returns the response message based on the response code.

Default options 

global $wp_version;
$args = array(
    'timeout'     => 5, // seconds
    'redirection' => 5, // how many times
    'httpversion' => '1.0',
    'user-agent'  => 'WordPress/' . $wp_version . '; ' . home_url(),
    'blocking'    => true, // php will stop until the request is processed. false to just ping but wp_remote_get will return false
    'headers'     => array(),
    'cookies'     => array(),
    'body'        => null,
    'compress'    => false,
    'decompress'  => true,
    'sslverify'   => true,
    'stream'      => false,
    'filename'    => null
);

3.16.20. Widget API wp-includes/widgets.php wp:api:widgets

3.16.20.1. dynamic_sidebar vs get_sidebar wp:f:dynamic_sidebar
Refer to wp:f:get_sidebar
get_sidebar looks for a template file which should have dynamic_sidebar('sidebar-slug')
3.16.20.2. register_sidebar wp:f:register_sidebar

Create a sidebar (widget area) where widget instances can be assigned to in wp admin UI. 

add_action( 'widgets_init', 'theme_slug_widgets_init' );
function theme_slug_widgets_init() {
    register_sidebar( array(
    'name' => __( 'Main Sidebar', 'theme-slug' ),
    'id' => 'sidebar-1',
    'description' => __( 'Widgets in this area will be shown on all posts and pages.', 'theme-slug' ),
    'class' => 'tal', // prepend so CSS result for WP Admin page becomes 'sidebar-tal', (default: empty)
    'before_widget' => '<li id="%1$s" class="widget %2$s">',
    'after_widget'  => '</li>',
    'before_title'  => '<h2 class="widgettitle">',
    'after_title'   => '</h2>',
    ) );
}
3.16.20.3. register_widget wp:f:register_widget
  • Used in wp:action:widgets_init
  • System widgets wp-includes/widgets/class-wp-widget-*.php and they're registered at wp-includes/widgets.php
<?php
// Register and load the widget
function wpb_load_widget() {
  register_widget( 'wpb_widget' );
}

add_action( 'widgets_init', 'wpb_load_widget' );

// Creating the widget 
class wpb_widget extends WP_Widget {

  function __construct() {
    // PHP 5.x
    //$this->WP_Widget( 'dokan-category-menu', 'Dokan: Product Category', $widget_ops );

    parent::__construct(

    // Base ID of your widget
  'wpb_widget',

  // Widget name will appear in UI
  __( 'WPBeginner Widget', 'wpb_widget_domain' ),

  // Widget description
  [ 'description' => __( 'Sample widget based on WPBeginner Tutorial', 'wpb_widget_domain' ), ]
    );
  }

  // Creating widget front-end

  public function widget( $args, $instance ) {
    $title = apply_filters( 'widget_title', $instance['title'] );

    // before and after widget arguments are defined by themes
    echo $args['before_widget'];
    if ( ! empty( $title ) ) {
  echo $args['before_title'] . $title . $args['after_title'];
    }

    // This is where you run the code and display the output
    echo __( 'Hello, World!', 'wpb_widget_domain' );
    echo $args['after_widget'];
  }

  // Widget Backend 
  public function form( $instance ) {
    if ( isset( $instance['title'] ) ) {
  $title = $instance['title'];
    } else {
  $title = __( 'New title', 'wpb_widget_domain' );
    }
    // Widget admin form
    ?>
    <p>
  <label
    for="<?php echo $this->get_field_id( 'title' ); ?>"><?php _e( 'Title:' ); ?></label>
  <input class="widefat"
         id="<?php echo $this->get_field_id( 'title' ); ?>"
         name="<?php echo $this->get_field_name( 'title' ); ?>"
         type="text" value="<?php echo esc_attr( $title ); ?>"/>
    </p>
    <?php
  }

  // Updating widget replacing old instances with new
  public function update( $new_instance, $old_instance ) {
    $instance          = [];
    $instance['title'] = ( ! empty( $new_instance['title'] ) ) ? strip_tags( $new_instance['title'] ) : '';

    return $instance;
  }
} // Class wpb_widget ends here


class MyNewWidget extends WP_Widget {

    function __construct() {
        // Instantiate the parent object

    }

    function widget( $args, $instance ) {
        // Widget output
    }

    function update( $new_instance, $old_instance ) {
        // Save widget options
    }

    function form( $instance ) {
        // Output admin widget options form
    }
}

function myplugin_register_widgets() {
    register_widget( 'MyNewWidget' );
}

add_action( 'widgets_init', 'myplugin_register_widgets' );

3.16.21. Navigation Menu Functions wp-includes/nav-menu.php,nav-menu-template.php

3.16.21.1. register_nav_menus, register_nav_menu, wp_nav_menu

wp:f:register_nav_menus used in wp:action:init or wp:action:after_setup_theme Register custom nav menus This is to create a theme location or navigation menu. Each theme location can have one menu created from wp-admin. Menu created from wp-admin can associate with multiple theme locations. The following registers 3 theme locations. Other translation plugins like Polylang can create theme locations for each extra language based on the already defined theme locations. 

function register_my_menus() {
  register_nav_menus(
    array(
  'main-menu' => __('Main Menu'),
  'about-menu' => __('About Menu'),
  'services-menu' => __('Services Menu')
    )
  );
}

Display in template

  • https://developer.wordpress.org/reference/functions/wp_nav_menu/
  • wp:filter:wp_nav_menu_args
    container
    'div' use what to wrap the menu, bool can be used
    menu_class
    'menu', css class, e.g. 'class-1 class-2'
    fallback_cb
    false call a cb function if menu doesn't exist e.g. 'WP_Bootstrap_Navwalker::fallback'
    walker
    object e.g. new WP_Bootstrap_Navwalker()
  • short-circuit
  • wp:filter:wp_nav_menu_container_allowedtags
  • wp:filter:wp_nav_menu_objects
  • wp:filter:wp_nav_menu_items
  • wp:filter:wp_nav_menu_$menu->slug_items
  • filter output
// already echo
wp_nav_menu( array( 'theme_location' => 'header-menu' ) );

3.16.22. Toolbar API wp-includes/admin-bar.php wp:api:toolbar

3.16.22.1. show_admin_bar wp:f:show_admin_bar

Use it in functions.php or wp:filter:show_admin_bar 

if ( ! current_user_can( 'manage_options' ) ) {
    show_admin_bar( false );
}

3.16.23. Pluggable functions wp-includes/pluggable.php wp:pluggable

  • These functions can be defined in plugins. If all active plugins don't define any of these functions, WP will define them
3.16.23.1. wp_get_current_user() wp:f:wp_get_current_user

  • Set wp:global:current_user if it's not set. If not logged in, set to 0 

    $current_user = wp_get_current_user();
if ( 0 == $current_user->ID ) {
// Not logged in.
} else {
// Logged in.
}
  • Also return wp:user
3.16.23.2. wp_mail() wp:f:wp_mail

  • Syntax 

    wp_mail( $to, $subject, $message, $headers = '', $attachments = array() )
  • Used filters

    • wp:filter:wp_mail_from

      • From Email address is first defined in $headers if it has From: your-email@a.ca
      • user@a.ca, Firstname Lastname <user@a.ca>

3.16.24. Administration API wp-admin/includes/admin.php wp:api:core admin

3.16.24.1. add_meta_box( $id, $title, $callback, $screen = null, $context = 'advanced', $priority = 'default', $callback_args = null ) wp:f:add_meta_box

  • Used in wp:action:add_meta_boxes

    • context
      advanced
      default
      side
      normal
    • Priority
      default
      default
      high
      right after title
      (no term)
      low
    /*
 * @param string                 $id            Meta box ID (used in the 'id' attribute for the meta box).
 * @param string                 $title         Title of the meta box.
 * @param callable               $callback      Function that fills the box with the desired content.
 *                                              The function should echo its output.
 * @param string|array|WP_Screen $screen        Optional. The screen or screens on which to show the box
 *                                              (such as a post type, 'link', or 'comment'). Accepts a single
 *                                              screen ID, WP_Screen object, or array of screen IDs. Default
 *                                              is the current screen.  If you have used add_menu_page() or
 *                                              add_submenu_page() to create a new screen (and hence screen_id),
 *                                              make sure your menu slug conforms to the limits of sanitize_key()
 *                                              otherwise the 'screen' menu may not correctly render on your page.
 * @param string                 $context       Optional. The context within the screen where the boxes
 *                                              should display. Available contexts vary from screen to
 *                                              screen. Post edit screen contexts include 'normal', 'side',
 *                                              and 'advanced'. Comments screen contexts include 'normal'
 *                                              and 'side'. Menus meta boxes (accordion sections) all use
 *                                              the 'side' context. Global default is 'advanced'.
 * @param string                 $priority      Optional. The priority within the context where the boxes
 *                                              should show ('high', 'low'). Default 'default'.
 * @param array                  $callback_args Optional. Data that should be set as the $args property
 *                                              of the box array (which is the second parameter passed
 *  
 *                                              to your callback). Default null.
 */
function add_meta_box( $id, $title, $callback, $screen = null, $context = 'advanced', $priority = 'default', $callback_args = null ) {

}
3.16.24.2. remove_meta_box( $id, $page, $context) wp:f:remove_meta_box
  • Return
  • string
    • authordiv
    • categorydiv
    • comemntstatusdiv
    • commentsdiv
    • formatdiv
    • pageparentdiv
    • postcustom
    • postexcerpt
    • postimagediv
    • revisionsdiv
    • slugdiv
    • submitdiv
    • tagsdiv-post_tag
    • tagsdiv-{$tax-name}
    • {$tax-name}div
    • trackbacksdiv
    • …
  • string. Type of screen
    • post
    • page
    • attachment
    • link
    • dashboard
    • any custom post type e.g. 'my-product'
  • string. 'normal', 'advanced', or 'side'
    Post edit screen
    normal, side and advanced
    Comments screen
    normal and side
    Menu meta boxes
    side
  • Return nothing
3.16.24.3. add_management_page( $page_title, $menu_title, $capability, $menu_slug, $function = '' )
  • Add sub menu to Tools admin menu

  • Used in wp:action:admin_menu 

    add_management_page(
 'Page title', // string
 'Menu title', // string
 'manage_options', // string. Capability that is required.
 'menu_slug', // string
 'callbackFunct', // string or array($this, 'options_page')
);

3.17. Default filters and actions wp:default filters

3.18. Filter

3.18.1. post_limits wp:filter:post_limits

For feed, use pre_option_posts_per_rss wp:filter:pre_option 

function lili_f_post_limits($limit, $query) {
 return 'LIMIT 0, 25';
 return $limit;
}

3.18.2. get_meta_sql wp:filter:get_meta_sql

add_filter( 'get_meta_sql', function($clause, $queries, $type, $primary_table, $primary_id_column, $context) {

    if (lili_request_cache('trigger_get_meta_sql')  === 'meta-query-archive-speaker-page') {
        // if the query is the query needed to be modified
        // echo "<pre>"; print_r($clause); echo "</pre>";
        // echo "<pre>"; print_r($context); echo "</pre>";
        $clause['where'] = '';

    }

    return $clause;
} );


// In template

// sort on custom fields which might not have values or exist
$args = array(
    'post_type'      => 'speaker',
    'posts_per_page' => - 1,
    'meta_query' => [
        'speaker_last_name_clause'     => [
            'key'     => 'speaker_last_name',
            'compare' => 'NOT EXISTS',
        ],
        'custom_order_clause'     => [
            'key'     => 'custom_order',
            'type'    => 'NUMERIC',
            'compare' => 'NOT EXISTS',
        ],
    ],
    'orderby'        => [
        'custom_order_clause'      => 'DESC',
        'speaker_last_name_clause' => 'ASC',
    ],
);

// use the trigger
lili_request_cache('trigger_get_meta_sql', 'meta-query-archive-speaker-page');

$speakers = new WP_Query($args);

// remove the trigger
lili_request_cache('trigger_get_meta_sql', '');

Sample 

[
    'join' => "LEFT JOIN wp_postmeta ON ( wp_posts.ID = wp_postmeta.post_id )  LEFT JOIN wp_postmeta AS mt1 ON (wp_posts.ID = mt1.post_id AND mt1.meta_key = 'speaker_last_name' )  LEFT JOIN wp_postmeta AS mt2 ON ( wp_posts.ID = mt2.post_id )  LEFT JOIN wp_postmeta AS mt3 ON (wp_posts.ID = mt3.post_id AND mt3.meta_key = 'custom_order' )",

    'where' => " AND ( 
  ( 
    wp_postmeta.meta_key = 'speaker_last_name' 
    OR
    mt1.post_id IS NULL
  )
  AND
  (
    mt2.meta_key= 'custom_order'
    OR
    mt3.post_id IS NULL
  )
)"
]

3.18.3. posts_clauses wp:filter:posts_clauses

  • Modify WP_Query object before it runs
  • $clauses = (array) apply_filters_ref_array( 'posts_clauses', array( compact( $pieces ), &$this ) );
function lili_posts_clauses($clauses, $query) {
  /*
  $clauses => [
    'where' => '...',
    'groupby' => '',
    'join' => '',
    'orderby' => 'wp_posts.post_date DESC',
    'distinct' => '',
    'fields' => 'wp_posts.*',
    'limits' => 'LIMIT 0, 20',
  ]
  $query is WP_Query object
  */
  // Identify a certain WP_Query
  $_s = lili_request_cache('trigger_posts_clauses');
  if ($_s == 'situationA') {
    $_old = $clauses['where'];
    $clauses['orderby'] = '';
    $_limit = explode(",",$clauses['limits']);
    $clauses['limits']='';
    $_limit_feature = end($_limit) - 3;
    reset($_limit);
    $_features_and_news = <<<SQL
SELECT * FROM
(SELECT p2.ID
FROM wp_posts p2
WHERE p2.post_type IN ('en-vedette')
ORDER BY p2.post_date DESC
LIMIT {$_limit_feature}
) AS sq1

UNION

SELECT * FROM
(SELECT p2.ID
FROM wp_posts p2
INNER JOIN wp_term_relationships tr ON tr.object_id = p2.ID
INNER JOIN wp_terms t ON t.term_id = tr.term_taxonomy_id
WHERE p2.post_type = 'nouvelles' AND t.slug = 'homepage-3-squares'
ORDER BY p2.post_date DESC
LIMIT 3) AS sq2
SQL;
    $clauses['where'] = $_old." AND wp_posts.ID IN ($_features_and_news)";
    // reset cache
    lili_request_cache('trigger_posts_clauses', '');
  }

  return $clauses;
}

3.18.4. query_vars wp:filter:query_vars

  • https://codex.wordpress.org/WordPress_Query_Vars
  • Used in wp:f:add_rewrite_tag
  • return query variable name in array not values
  • class-wp.php $public_query_vars $private_query_vars

  • mysite.ca/?p=123 where p is a public query var where it can be used in URL and has filter effect while private query var can only be used in PHP: 

    $query = new WP_Query(array(
    'post__in' => array(3, 7) // post__in is private
));

3.18.5. request wp:filter:request

  • filter the array of parsed query variables
  • apply_filters( 'request', array $query_vars )
  • empty array
  • [ 'post_type' => 'mycpt' ]
  • [ 'page' => '', 'mycpt' => 'mypost-slug', 'post_type' => 'mycpt', 'name' => 'mypost-slug' ]

3.18.6. page_link, post_link wp:filter:page_link wp:filter:post_link

The filter is used in get_permalink 

// When set to true, a structural link will be returned, rather than the actual URI. Example: http://www.example.com/%postname% 
// instead of http://www.example.com/my-post

function append_query_string( $url, $post, $leavename=false ) {

  if ( $post->post_type == 'post' ) {
    $url = add_query_arg( 'foo', 'bar', $url );
  }
  return $url;
}
add_filter( 'post_link', 'append_query_string', 10, 3 );

// add custom query var
function myplugin_register_query_vars( $vars ) {
    $vars[] = 'key1';
    $vars[] = 'key2';
    return $vars;
}
add_filter( 'query_vars', 'myplugin_register_query_vars' );

// use custom query var later

function myplugin_pre_get_posts( $query ) {
    // check if the user is requesting an admin page 
    // or current query is not the main query
    if ( is_admin() || ! $query->is_main_query() ){
        return;
    }
    $meta_query = array();
    // add meta_query elements
    if( !empty( get_query_var( 'key1' ) ) ){
        $meta_query[] = array( 'key' => 'key1', 'value' => get_query_var( 'key1' ), 'compare' => 'LIKE' );
    }
    if( !empty( get_query_var( 'key2' ) ) ){
        $meta_query[] = array( 'key' => 'key2', 'value' => get_query_var( 'key2' ), 'compare' => 'LIKE' );
    }
    if( count( $meta_query ) > 1 ){
        $meta_query['relation'] = 'AND';
    }
    if( count( $meta_query ) > 0 ){
        $query->set( 'meta_query', $meta_query );
    }
}
add_action( 'pre_get_posts', 'myplugin_pre_get_posts', 1 );

3.18.7. url_to_postid

url_to_postid can take full url. But it may not work for custom post type even though get_permalink and rewrite works.

When URL has url parameter p or other special parameters, it will directly return the post id. 

echo url_to_postid(get_permalink()); // the current post id.

function lili_get_post_id($custom_permalink) {
  global $wpdb;
  $post_id = $wpdb->get_col($wpdb->prepare("SELECT post_id FROM $wpdb->postmeta WHERE meta_key= 'custom_permalink' AND meta_value = '%s';", $custom_permalink ));
  return $post_id[0];
}

function lili_url_to_postid($url) {
  $url_parse = wp_parse_url($url);
  if ($url_parse !== false && !empty($url_parse['path'])) {
    $post_id = lili_get_post_id(ltrim(strtolower($url_parse['path']), '/'));
    if ($post_id) {
      $post_id = (int) $post_id;
      return '?p='.$post_id;
    }
  }
  return $url;
}
add_filter( 'url_to_postid', 'lili_url_to_postid', 10, 1);

3.18.8. pre_option_(option_name) wp:filter:pre_option_*

Temporarily alter an option without changing in database.

3.18.9. found_posts wp:filter:found_posts

Modify number of found posts 

add_filter( 'found_posts', 'myprefix_adjust_offset_pagination', 1, 2 );
function myprefix_adjust_offset_pagination($found_posts, $query) {

    //Define our offset again...
    $offset = 10;

    //Ensure we're modifying the right query object...
    if ( $query->is_home() ) {
    //Reduce WordPress's found_posts count by the offset... 
    return $found_posts - $offset;
    }
    return $found_posts;
}

3.18.10. template_include wp:filter:template_include

function pubx_cuw_template_include($template) {
  global $wp;
  try {
    $_url = explode('/',$wp->request);
    if (!empty($_url) && count($_url) == 3) {
  $adunit=$_url[1];
  $adsize=$_url[2];
  $dfp = array(
    'em_CUW_BB_1' => ['95740733/'.$adunit, '300x250'],
    'em_CUW_BB_2' => ['95740733/'.$adunit, '300x250'],
    'em_CUW_BB_3' => ['95740733/'.$adunit, '300x250'],
    'em_CUW_LB_1' => ['95740733/'.$adunit, $adsize], // 640x90, 300x90
    'em_CUW_LB_2' => ['95740733/'.$adunit, $adsize], // 640x90, 300x90
  );

  if (isset($dfp[$adunit]) && strlen($adsize) < 10) {
    $_ad_unit = $dfp[$adunit][0];
    $_ad_size = $dfp[$adunit][1];
    $htmlFile = <<<HTML
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Canadian Underwriter Partnership</title>
<script src="//code.jquery.com/jquery-1.10.2.js"></script>
</head>
<body>
  <script>
    var rnd=Math.floor(Math.random() * (999999 - 10 +1)) + 10;
    var theUrl="//pubads.g.doubleclick.net/gampad/adx?iu=/{$_ad_unit}&sz={$_ad_size}&c=" + rnd;
    var xmlHttp = null;
    xmlHttp = new XMLHttpRequest();
    xmlHttp.open( "GET", theUrl );
    xmlHttp.send();
    xmlHttp.onreadystatechange = function() {
  if (xmlHttp.readyState==4 && xmlHttp.status==200) {
    var xmlDoc = $.parseHTML( xmlHttp.response );
    var adlink = $(xmlDoc).find('a').attr('href');
    if (typeof adlink == "undefined") adlink = "//www.example.com";
    window.location = adlink;
  }
    }
  </script>
</body>
</html>
HTML;

    global $wp_query;
    $wp_query->is_404 = false;
    status_header(200);
    echo $htmlFile;
    $new_template = locate_template( ['empty_page.php'] );
    return $new_template;
  }
    }
    elseif (other_situation) {
  // you can just flush a buffer or output something 
  // then set some headers
  // and a response code
  // then exit();
    }
  }
  catch (Exception $e) {
    return $template;
  }

  return $template;

}
add_filter('template_include', 'pubx_cuw_template_include',99);

3.18.11. flush_rewrite_rules_hard wp:filter:flush_rewrite_rules_hard

Stop wp to modify .htaccess inside the Wordpress block 

add_filter( 'flush_rewrite_rules_hard', '__return_false' );

// __return_false is a wp function

// remove the filter 
remove_filter( 'flush_rewrite_rules_hard', 'filter_flush_rewrite_rules_hard', 10, 1 );

3.18.12. tiny_mce_before_init wp:filter:tiny_mce_before_init

Change TinyMCE settings https://codex.wordpress.org/TinyMCE Default setting https://www.tinymce.com/docs/configure/ 

add_filter( 'tiny_mce_before_init', 'myformatTinyMCE' );
function myformatTinyMCE( $in ) {
  $in['wordpress_adv_hidden'] = FALSE; // enable advanced edit mode

  //region elements that are not included will be removed. Add all elements.
  $opts = '*[*]';
  $in['valid_elements'] = $opts;
  $in['extended_valid_elements'] = $opts;
  //endregion

  //region To ensure p tag is not removed when switching between Visual and Text modes
  $in['wpautop'] = false;
  $in['indent'] = true;
  //endregion

  //region p tag will not be added for any new line.
  // To create a paragraph in Visual mode, Shift+Enter. Otherwise, enter will simply create a <br>
  // This helps TinyMCE wraps shortcode on any line with p tag.
  $in['force_p_newlines'] = false;
  $in['forced_root_block'] = '';
  //endregion

  return $in;
}

3.18.13. the_title wp:filter:the_title

  • Used in wp:f:get_the_title
  • Built-in filters
    wp:f:wptexturize
    replace some text to HTML entities but skip the replacement in some HTML tags pre, code, kdb, style, script, tt
    wp:f:convert_chars
    deal with & to ensure HTML entities are correct
    (no term)
    trim
add_filter( 'the_title', function( $title, $id ) {
    // $id The post ID
    return $title;
},10,2);

3.18.14. the_title_rss wp:filter:the_title_rss

add_filter( 'the_title_rss', function( $title ) {
    return $title;
},10);

3.18.15. the_content wp:filter:the_content

Refer to 

add_filter( 'the_content', 'wpautop');
// remove_filter( 'the_content', 'wpautop');
add_filter( 'the_content', 'shortcode_unautop'  );

//Insert ads after second paragraph of single post content.

add_filter( 'the_content', 'prefix_insert_post_ads' );

function prefix_insert_post_ads( $content ) {

    $ad_code = '<div>Ads code goes here</div>';

    if ( is_single() && ! is_admin() ) {
    return prefix_insert_after_paragraph( $ad_code, 2, $content );
    }

    return $content;
}

// Parent Function that makes the magic happen

function prefix_insert_after_paragraph( $insertion, $paragraph_id, $content ) {
    $closing_p = '</p>';
    $paragraphs = explode( $closing_p, $content );
    foreach ($paragraphs as $index => $paragraph) {

    if ( trim( $paragraph ) ) {
        $paragraphs[$index] .= $closing_p;
    }

    if ( $paragraph_id == $index + 1 ) {
        $paragraphs[$index] .= $insertion;
    }
    }

    return implode( '', $paragraphs );
}

3.18.16. upload_mimes wp:filter:upload_mimes

Add mime type so that wp can handle its content-type header to browser for upload and sending. 

function cc_mime_types($mimes) {
  $mimes['svg'] = 'image/svg+xml';
  return $mimes;
}
add_filter('upload_mimes', 'cc_mime_types');

3.18.17. widget_text wp:filter:widget_text

Display wp:f:add_shortcode in widget. Filter content output 

function exam_plug_text_replace($content) {
  $new_content = '';
  $new_content.= $content . ':)';
  return $new_content;
}
add_filter('widget_text', 'exam_plug_text_replace');

3.18.18. sidebars_widgets wp:filter:sidebars_widgets

Filters the list of sidebars and their widgets. Changing this will change the widget registeration on db. 

function disable_all_widgets($sidebars_widgets) {
  //var_dump($sidebars_widgets);
  // disable all widget areas
  // $sidebars_widgets = array(false);
  $sidebars_widgets['sidebar-id-1'] = ['widget_name-1'];
  return $sidebars_widgets;
}
add_filter('sidebars_widgets', 'disable_all_widgets');

In order to have an instance of a widget, the easiest way is to create another sidebar with the widget in wp-admin

3.18.19. show_admin_bar wp:filter:show_admin_bar

wp:f:show_admin_bar 

add_filter('show_admin_bar', '__return_false');

3.18.20. excerpt_more wp:filter:excerpt_more

  • Default is echo '[…]'
function twentyseventeen_excerpt_more( $link ) {
    if ( is_admin() ) {
        return $link;
    }

    $link = sprintf( '<p class="link-more"><a href="%1$s" class="more-link">%2$s</a></p>',
        esc_url( get_permalink( get_the_ID() ) ),
        /* translators: %s: Name of current post */
        sprintf( __( 'Continue reading<span class="screen-reader-text"> "%s"</span>', 'twentyseventeen' ), get_the_title( get_the_ID() ) )
    );
    return ' &hellip; ' . $link;
}
add_filter( 'excerpt_more', 'twentyseventeen_excerpt_more' );

3.18.21. excerpt_length wp:filter:excerpt_length

55
Default number of words

3.18.22. wp_calculate_image_srcset wp:filter:wp_calculate_image_srcset

$sources = apply_filters( 'wp_calculate_image_srcset', $sources, $size_array, $image_src, $image_meta, $attachment_id );

// sample
$sources = [
    '300' => [
        'url' => 'https://a.ca/wp-content/uploads/2019/03/image-name-100x100.jpg',
        'descriptor' => 'w',
        'value' => '300'
    ],
    '930' => [
        'url' => 'https://a.ca/wp-content/uploads/2015/12/Harbour-House-Sign.jpg',
        'descriptor' => 'w',
        'value' => '930'

    ],
    // ...
];

3.18.23. wp_calculate_image_srcset_meta wp:filter:wp_calculate_image_srcset_meta

add_filter( 'wp_calculate_image_srcset_meta', function($image_meta, $size_array, $image_src, $attachment_id ) {
    //var_dump($image_meta);

    if (isset($image_meta['image_meta'])
    && isset($image_meta['image_meta']['cpt_last_cropping_data'])
    && isset($image_meta['image_meta']['cpt_last_cropping_data']['date'])) {
        foreach ( $image_meta[ 'sizes' ] as $k => $v ) {
        //var_dump($v);
        if (isset($v['file']) && $v['file']) {
            $image_meta[ 'sizes' ][$k]['file'] .= '?lastcrop='.$image_meta['image_meta']['cpt_last_cropping_data']['date'];
        }
        }
    }

    return $image_meta;
}, 11, 4);

3.18.24. wp_calculate_image_sizes wp:filter:wp_calculate_image_sizes

// wp_calculate_image_sizes( array|string $size, string $image_src = null, array $image_meta = null, int $attachment_id )

function twentyseventeen_content_image_sizes_attr( $sizes, $size ) {
    $width = $size[0];

    if ( 740 <= $width ) {
        $sizes = '(max-width: 706px) 89vw, (max-width: 767px) 82vw, 740px';
    }

    if ( is_active_sidebar( 'sidebar-1' ) || is_archive() || is_search() || is_home() || is_page() ) {
        if ( ! ( is_page() && 'one-column' === get_theme_mod( 'page_options' ) ) && 767 <= $width ) {
             $sizes = '(max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px';
        }
    }

    return $sizes;
}
add_filter( 'wp_calculate_image_sizes', 'twentyseventeen_content_image_sizes_attr', 10, 2 );

3.18.25. wp_resource_hints

apply_filters( 'wp_resource_hints', array $urls, string $relation_type )

relation_type :: dns-prefetch, preconnect, prefetch, and prerender

https://make.wordpress.org/core/2016/07/06/resource-hints-in-4-6/ 

function twentyseventeen_resource_hints( $urls, $relation_type ) {
        if ( wp_style_is( 'twentyseventeen-fonts', 'queue' ) && 'preconnect' === $relation_type ) {
                $urls[] = array(
                        'href' => 'https://fonts.gstatic.com',
                        'crossorigin',
                );
        }

        return $urls;
}
add_filter( 'wp_resource_hints', 'twentyseventeen_resource_hints', 10, 2 );

3.18.26. script_loader_src style_loader_src wp:filter:script_loader_src wp:filter:style_loader_src

Remove current site url from script and style files 

add_filter( 'script_loader_src', 'wpse47206_src' );
add_filter( 'style_loader_src', 'wpse47206_src' );
function wpse47206_src( $url ) {
        if( is_admin() ) return $url;
        return str_replace( site_url(), '', $url );
}

3.18.27. imgae_size_names_choose wp:filter:image_size_names_choose

add_image_size( 'video-thumbnail', 320, 180, true );

add_filter( 'image_size_names_choose', function ( $sizes ) {
    return array_merge( $sizes, array(
        'video-thumbnail' => __( 'Video Thumbnail' ),
    ) );
});

3.18.28. get_header_image_tag

apply_filters( 'get_header_image_tag', string $html, object $header, array $attr );

function twentyseventeen_header_image_tag( $html, $header, $attr ) {
    if ( isset( $attr['sizes'] ) ) {
        $html = str_replace( $attr['sizes'], '100vw', $html );
    }
    return $html;
}
add_filter( 'get_header_image_tag', 'twentyseventeen_header_image_tag', 10, 3 );

In template 

<img src="<?php header_image(); ?>" height="<?php echo get_custom_header()->height; ?>" width="<?php echo get_custom_header()->width; ?>" alt="" />

3.18.29. wp_nav_menu_args wp:filter:wp_nav_menu_args

// $args = apply_filters( 'wp_nav_menu_args', $args );

add_filter( 'wp_nav_menu_args', 'bfg_nav_menu_args_filter' );
function bfg_nav_menu_args_filter( $args ) {

    require_once( BFG_THEME_MODULES . 'class-wp-bootstrap-navwalker.php' );

    if ( 'primary' === $args['theme_location'] ) {
    $args['container'] = false;
    $args['menu_class'] = 'navbar-nav mr-auto';
    $args['fallback_cb'] = 'WP_Bootstrap_Navwalker::fallback';
    $args['walker'] = new WP_Bootstrap_Navwalker();
    }
    return $args;
}

3.18.30. wp_get_attachment_image_src wp:filter:wp_get_attachment_image_src

/**
 * Add URL Parameter lastcrop with current datetime to bust image cache
 *
 * Add cache busting if image has metadata cpt_last_cropping_data which is added in filter gncm_crop_thumbnails_before_update_metadata
 *
 * @param $image
 * @param $attachment_id
 * @param $size
 * @param $icon
 *
 * @return array
 */
function gncm_wp_get_attachment_image_src( $image, $attachment_id, $size, $icon ) {
    if ( is_array( $image ) && ! empty( $image )
         && ! is_admin()
         && function_exists( 'http_build_url' ) ) {
        $url  = @parse_url( $image[0] );
        $meta = wp_get_attachment_metadata( $attachment_id );
        if ( ! empty( $url ) && ! empty( $meta )
             && isset( $meta['image_meta'] )
             && isset( $meta['image_meta']['cpt_last_cropping_data'] )
             && isset( $meta['image_meta']['cpt_last_cropping_data']['date'] )
        ) {
            $url['query'] = ( isset( $url['query'] ) ) ? $url['query'] : '';
            parse_str( $url['query'], $params );
            $params       = array_merge( [
                'lastcrop' => $meta['image_meta']['cpt_last_cropping_data']['date']
            ], $params );
            $url['query'] = http_build_query( $params );
            $image[0]     = http_build_url( $url );
        }
    }

    return $image;
}
add_filter( 'wp_get_attachment_image_src', 'gncm_wp_get_attachment_image_src', 11, 4 );

3.18.31. wp_get_attachment_image_attributes

  • apply_filters( 'wp_get_attachment_image_attributes', array $attr, WP_Post $attachment, string|array $size )
  • (array) Attributes for the image markup.
  • (WP_Post) Image attachment post.
  • (string|array) Requested size. Image size or array of width and height values (in that order). Default 'thumbnail'.
function twentyseventeen_post_thumbnail_sizes_attr( $attr, $attachment, $size ) {
    if ( is_archive() || is_search() || is_home() ) {
        $attr['sizes'] = '(max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px';
    } else {
        $attr['sizes'] = '100vw';
    }

    return $attr;
}
add_filter( 'wp_get_attachment_image_attributes', 'twentyseventeen_post_thumbnail_sizes_attr', 10, 3 );

3.18.32. widget_tag_cloud_args

function twentyseventeen_widget_tag_cloud_args( $args ) {
        $args['largest']  = 1;
        $args['smallest'] = 1;
        $args['unit']     = 'em';
        $args['format']   = 'list';

        return $args;
}
add_filter( 'widget_tag_cloud_args', 'twentyseventeen_widget_tag_cloud_args' );

3.18.33. comments_open

wp:plugin:disable-comments

3.18.34. deprecated_constructor_trigger_error wp:filter:deprecated_constructor_trigger_error

add_filter('deprecated_constructor_trigger_error', '__return_false');

3.19. Action

3.19.1. activated_plugin wp:action:activated_plugin

  • Change plugins loading order. By default is alpha based on plugin's path + plugin's php file
  • It reads and updates wp:db:wp_options:active_plugins
  • Use the following code at the beginning of a plugin to change the order
function this_plugin_last() {
    $wp_path_to_this_file = preg_replace('/(.*)plugins\/(.*)$/', WP_PLUGIN_DIR."/$2", __FILE__);
    $this_plugin = plugin_basename(trim($wp_path_to_this_file));
    $active_plugins = get_option('active_plugins');
    $this_plugin_key = array_search($this_plugin, $active_plugins);
    array_splice($active_plugins, $this_plugin_key, 1);
    array_push($active_plugins, $this_plugin);
    update_option('active_plugins', $active_plugins);
}
add_action("activated_plugin", "this_plugin_last");

3.19.2. plugins_loaded wp:action:plugins_loaded

  • default added actions
    • wp_maybe_load_widgets
    • wp_maybe_load_embeds
    • _wp_customize_include

3.19.3. after_setup_theme wp:action:after_setup_theme

Called each page load after the theme is initialized. Do basic setup, registration and init actions for a theme

May run these in this action

3.19.4. init wp:action:init

3.19.5. widgets_init wp:action:widgets_init

Refer to wp:filter:sidebars_widgets to modify registered sidebars and their widgets. Refer to wp:f:register_widget to create a widget 

function lili_widgets_init() {
  $args = array(
    'name'          => __( 'Sidebar name', 'theme_text_domain' ),
    'id'            => 'unique-sidebar-id', // %1$s
    'description'   => '',
    'class'         => '', // %2$s, will have ~sidebar-~ prepended
    'before_widget' => '<li id="%1$s" class="widget %2$s">',
    'after_widget'  => '</li>',
    'before_title'  => '<h2 class="widgettitle">',
    'after_title'   => '</h2>' );
  register_sidebar( $args );

  // register_sidebar( $args2 );

  // Create multiple sidebars with the same config
  $args = array(
    'name'          => __('Sidebar %d'),
    'id'            => 'sidebar',          
    'description'   => '',
    'class'         => '',
    'before_widget' => '<li id="%1$s" class="widget %2$s">',
    'after_widget'  => '</li>',
    'before_title'  => '<h2 class="widgettitle">',
    'after_title'   => '</h2>' );

  // register_sidebars( 2, $args );

}
add_action('widgets_init', 'lili_widgets_init');

Display 

<?php if ( is_active_sidebar( 'home_right_1' ) ) : ?>
  <div id="primary-sidebar" class="primary-sidebar widget-area" role="complementary">
    <?php dynamic_sidebar( 'home_right_1' ); ?>
  </div><!-- #primary-sidebar -->
<?php endif; ?>

3.19.6. admin_menu wp:action:admin_menu

3.19.7. admin_init wp:action:admin_init

It's triggered before any other hook when a user accesses the admin area. This hook doesn't provide any parameters, so it can only be used to callback a specified function.

These core functions are called: register_admin_color_schemes send_frame_options_header _wp_check_for_scheduled_split_terms _wp_admin_bar_init _maybe_update_core _maybe_update_plugins _maybe_update_themes

3.19.8. parse_request wp:action:parse_request

3.19.9. pre_get_posts wp:action:pre_get_posts

  • Provide a chance to modify $query object (or do something else) before it's run. $query can be any main (often) or secondary query
  • Do not use this to alter query for single Page requests
  • Add action in functions.php. Query is already made when template is loaded
  • do_action_ref_array( 'pre_get_posts', array( &$this ) )
  • Use it instead of query_posts
  • Refer to WP_Query
add_action( 'pre_get_posts', 'lili_pre_get_posts' );
// Should not call this in template because in template the $query is already run
function lili_pre_get_posts( $query ) {
  if ( ! is_page() && ! is_home() 
   && $query->is_main_query() ) {
    $query->set( 'post_type', array( 'post', 'movies' ) );
  }
  // this function should return nothing.
}

3.19.10. send_headers wp:action:send_headers

$regex_path_patterns = array(
  '#^/news/?#',
  '#^/about/?#',
);

// Loop through the patterns.
foreach ($regex_path_patterns as $regex_path_pattern) {
  if (preg_match($regex_path_pattern, $_SERVER['REQUEST_URI'])) {
    add_action( 'send_headers', 'add_header_nocache', 15 );

    // No need to continue the loop once there's a match.
    break;
  }
}
function add_header_nocache() {
  header( 'Cache-Control: no-cache, must-revalidate, max-age=0' );
}


/* For WP REST API specific paths, we use a different approach by using the rest_post_dispatch filter */

// wp-json paths or any custom endpoints
$regex_json_path_patterns = array(
  '#^/wp-json/wp/v2?#',
  '#^/wp-json/?#'
);

foreach ($regex_json_path_patterns as $regex_json_path_pattern) {
  if (preg_match($regex_json_path_pattern, $_SERVER['REQUEST_URI'])) {
  // re-use the rest_post_dispatch filter in the Pantheon page cache plugin
  add_filter( 'rest_post_dispatch', 'filter_rest_post_dispatch_send_cache_control', 12, 2 );

  // Re-define the send_header value with any custom Cache-Control header
  function filter_rest_post_dispatch_send_cache_control( $response, $server ) {
      $server->send_header( 'Cache-Control', 'no-cache, must-revalidate, max-age=0' );
      return $response;
  }
  break;
  }
}

3.19.11. wp_enqueue_scripts wp:action:wp_enqueue_scripts

Can use wp:f:wp_enqueue_style and wp:f:wp_enqueue_script. Priority might be needed to change to greater 10 to load after default theme/style.css Use wp:filter:script_loader_src and wp:filter:style_loader_src to change the source of scripts and styles 

function lili_scripts() {
  wp_enqueue_style('lili-custom-styles', get_stylesheet_directory_uri().'/css/style.css');
}
add_action( 'wp_enqueue_scripts', 'lili_scripts', 20);

3.19.12. admin_head wp:action:admin_head

function my_custom_admin_head() {
  echo '<style>[for="wp_welcome_panel-hide"] {display: none !important;}</style>';
}
add_action( 'admin_head', 'my_custom_admin_head' );

// Add inline JS in the admin head with the <script> tag

function my_custom_admin_head() {
  echo '<script type=""text/javascript">console.log('admin script')</script>';
}
add_action( 'admin_head', 'my_custom_admin_head' );

3.19.13. template_redirect wp:action:template_redirect

Refer to wp:php:redirect

3.19.14. wp_head, wp_footer, get_footer wp:action:wp_head

  • These actions are good for adding inline CSS and JavaScripts
  • Refer to wp:default filters
  • wp_head(); and wp_footer(); are usually called in template files header/footer.php
    default wp_footer actions
    wp_print_footer_scripts p:20 and wp_admin_bar_render p:1000
  • get_footer action is called at the beginning of the get_footer(); function, which will include footer.php file later

If a parameter is passed, footer-[$name].php is also included and you can play around $name in the get_footer hook like this 

<?php
function themeslug_footer_hook( $name ) {
    if ( 'new' == $name ) { ?>
        <script>
            (function($) {
                //put all your jQuery goodness in here.
            })(jQuery);
        </script>
    <?php
    }
  // you can also print
  if ( is_singular() && pings_open() ) {
    printf( '<link rel="pingback" href="%s">' . "\n", get_bloginfo( 'pingback_url' ) );
  }

}
add_action( 'get_footer', 'themeslug_footer_hook' );

Remove emoji javascript and style 

remove_action('wp_head', 'print_emoji_detection_script', 7);

remove_action('wp_print_styles', 'print_emoji_styles');

remove_action('wp_head', 'wlwmanifest_link'); // Windows Live Writer support

remove_action('wp_head', 'wp_generator'); // wp version

remove_action ('wp_head', 'rsd_link'); // remove EditURI/RSD Really Simple Directory and xmlrpc.php

remove_action( 'wp_head', 'wp_shortlink_wp_head'); // e.g. http://yoursite.com/?p=7

// By default, wp:f:add_theme_support automatic-feed-links is not enabled and the feed links are not inserted 
remove_action( 'wp_head', 'feed_links', 2 ); // e.g. <link rel="alternate" type="application/rss+xml" title="WP Site &raquo; Feed" href="http://localhost/wp/feed/" />
remove_action('wp_head', 'feed_links_extra', 3 ); // remove comment feeds e.g. Like: <link rel="alternate" type="application/rss+xml" title="WP Site &raquo; Comments Feed" href="http://localhost/wp/comments/feed/" />

remove_action('wp_head', 'adjacent_posts_rel_link'); // remove PREV and NEXT links e.g. <link rel='prev' title='The Post Before This One' href='http://localhost/wp/?p=4' />

remove_action( 'wp_head','rest_output_link_wp_head'); // wp-json

3.19.15. add_meta_boxes, add_meta_boxespost_type wp:action:add_meta_boxes

  • lili_add_meta_boxes($post_type, $post) takes 2 arguments, no return
  • lili_add_meta_boxes_{post_type}($post) takes 1 argument, no return
  • Use
  • or modify global $wp_meta_boxes
  • $wp_meta_boxes['post-type']['normal']['core']['postexcerpt']['title'] = 'Your title';
  • You can change the title, callback (complete change to another function) and different context.
  • The callback most likely outputs a label, form field and some description.
  • You can either completely change the callback or use wp:filter:gettext to change the translation

3.19.16. edit_page_form wp:action:edit_page_form

Fires after 'normal' context meta boxes have been output for the 'page' post type. Control admin Edit Page form.

wp:f:wp_editor 

function volunteer_hours_editor() {
  $content = get_post_meta(get_the_ID(), 'volunteer-hours', true);
  $settings = array('textarea_name' => 'volunteer-hours');
  //wpautop($content);
  wp_editor( $content, 'volunteer-hours', $settings);
}

3.19.17. edit_form_advanced

wp:action:edit_form_advanced Same as wp:action:edit_page_form but for all post types other than 'page'

3.19.18. save_post wp:action:save_post

  • Called when a post or page is created or updated, which could be from an import, post/page edit form, xmlrpc, or post by email
  • The data for the post is stored in $_POST, $_GET or global $post_data, depending on how the post was edited. e.g. quick edits use $_POST
  • Use 
add_action( 'save_post', 'save_post', 10, 2 );
function save_post( $post_id, $post ) {
    if ( !current_user_can( 'edit_post', $post_id ) ) {
        return $post_id; // end the action. Not necessary to return.
    }

    // update a custom field
    update_post_meta( $post_id, 'volunteer-hours', stripslashes( wpautop($_POST['volunteer-hours']) ) );

    // auto select parent term if child term is selected for a post type
    if ($post->post_type == 'product') {
        $tax = 'product_category';
        $terms = wp_get_post_terms($post_id, $tax);
        foreach ($terms as $term) {
            while ($term->parent != 0 && !has_term( $term->parent, $tax, $post )) {
                // move upward until we get to 0 level terms
                // if parent is already selected, don't add
                wp_set_post_terms($post_id, array($term->parent), $tax, true);
                $term = get_term($term->parent, $tax);
            }
        }
    }
    // auto select parent term.

}

3.19.19. do_feed_$feedname wp:action:do_feed_$feedname

3.20. FTP setting wp:ftp

WordPress is usually run using Apache or Nginx user which might not have permission to write files. docker container map /host/path/to/html:/var/www Setup these in wp-config.php 

define('FTP_BASE', '/host/path/to/html/');
define('FS_METHOD', 'direct');
define('FTP_HOST', '8.8.8.8'); // public ip
define('FTP_USER', 'ftpuser');
define('FTP_PASS', 'ftppassword');
define('FTP_SSL', true);

https://codex.wordpress.org/Editing_wp-config.php#WordPress_Upgrade_Constants outputs comment form

3.21. Sanitizing and Escaping User Data

https://codex.wordpress.org/Validating_Sanitizing_and_Escaping_User_Data

Sanitize data before it's saved 

<input type="text" id="title" name="title" />

$title = sanitize_text_field( $_POST['title'] );
update_post_meta( $post->ID, 'title', $title );

sanitize_email()
sanitize_file_name()
sanitize_html_class()
sanitize_key()
sanitize_meta()
sanitize_mime_type()
sanitize_option()
sanitize_sql_orderby()
sanitize_text_field()
sanitize_textarea_field()
sanitize_title()
sanitize_title_for_query()
sanitize_title_with_dashes()
sanitize_user()

Escape

Translate then escape
esc_html__( $text, $domain = 'default')
Escape for HTML element attribute value
esc_attr
Escape js for HTML element attribute value
esc_js
(no term)
Escape for <textarea> value
<h4><?php echo esc_html( $title ); ?></h4>
<img src="<?php echo esc_url( $great_user_picture_url ); ?>" />
<a href="#" onclick="<?php echo esc_js( $custom_js ); ?>">Click me</a>
<ul class="<?php echo esc_attr( $stored_class ); ?>">
<textarea><?php echo esc_textarea( $text ); ?></textarea>

3.22. Ajax wp:ajax

3.22.1. Ajax on wp admin pages

Javascript global variable ajaxurl is defined on admin pages

functions.php 

add_action( 'admin_enqueue_scripts', 'add_ajax_file');
function add_ajax_file() {
  wp_enqueue_script( 'ajax_custom_script', plugins_url( '/js/my.js' , __FILE___), array('jquery') );
}

add_action( 'wp_ajax_my_action', 'my_action');

function my_action() {
  global $wpdb;
  $data = intval($_POST['data']);
  echo $data;
  // echo json_encde($data);
  wp_die();
}

my.js 

jQuery(document).ready(function($) {
  var data = {
    'action': 'my_action',
    'data': 'some data'
  }
  $.post(ajaxurl, data, function(r) {
    console.log('response:', r);
  });
})

3.22.2. Ajax on wp front end pages

Define global variable mynamespace.ajaxurl functions.php 

add_action( 'wp_enqueue_scripts', 'add_custom_namespace' );

function add_custom_namespace() {
  wp_localize_script( 'jquery', 'mynamespace', 
   array(
    'ajaxurl' => admin_url('admin-ajax.php'),
    'data' => 'some data'
   ) 
  );
}

my.js 

jQuery(document).ready(function($) {
  var data = {
    'action': 'my_action',
    'data': mynamespace.data
  }
  $.post(mynamespace.ajaxurl, data, function(r) {
    console.log('response:', r);
  });
})

functions.php 

add_action( 'wp_ajax_my_action', 'my_action');
add_action( 'wp_ajax_nopriv_my_action', 'my_action'); // non-logged in users

// same my_action

3.23. Theme wp:theme

3.23.1. child theme

https://codex.wordpress.org/Child_Themes wp-content/themes/parent-theme-name-child wp-content/themes/parent-theme-name-child/style.css

For new theme, index.php is required 

/*
 Theme Name:   Twenty Fifteen Child
 Theme URI:    http://example.com/twenty-fifteen-child/
 Description:  Twenty Fifteen Child Theme
 Author:       John Doe
 Author URI:   http://example.com
 Template:     twentyfifteen
 Version:      1.0.0
 License:      GNU General Public License v2 or later
 License URI:  http://www.gnu.org/licenses/gpl-2.0.html
 Tags:         light, dark, two-columns, right-sidebar, responsive-layout, accessibility-ready
 Text Domain:  twenty-fifteen-child
*/

Only Theme Name and Template (the name of the parent theme directory) are required

functions.php in child loads first then the parent's 

add_action( 'wp_enqueue_scripts', 'my_theme_enqueue_styles' );
function my_theme_enqueue_styles() {
  $parent_style = 'parent-style'; // This is 'twentyfifteen-style' for the Twenty Fifteen theme.

  wp_enqueue_style( $parent_style, get_template_directory_uri() . '/style.css' );

  // child theme style.css is loaded after parent's style.css
  wp_enqueue_style( 'child-style',
    get_stylesheet_directory_uri() . '/style.css',
    array( $parent_style ),
    wp_get_theme()->get('Version')
  );
}

That requires parent functions.php to have 

if ( ! function_exists( 'theme_special_nav' ) ) {
    function theme_special_nav() {
    //  Do something.
    }
}

3.23.2. functions.php wp:theme:functions

3.23.3. Template Files

single.php single-{post-type}.php archive.php archive-{post-type}.php

if single.php and archive.php don't exist, wp looks for index.php

3.23.4. Page template

  • Hierarchy
    Global Custom Page Template
    can be applied to multiple pages. Set in Edit Post > Page Attributes > Template
    page-{slug}.php
    for one specific page. Directly put them in theme folder
    page-{id}.php
    for one specific page. Directly put them in theme folder
    page.php
    refer to conditional tags
    (no term)
    singular.php
    (no term)
    index.php
  • Attachment

    • MIME_type.php

      • image.php, video.php, application.php
      • For text/plain, in order
      • text_plain.php
      • plain.php
      • text.php
    • attachment.php
    • single-attachment.php
    • single.php
    • singular.php
3.23.4.1. Global Custom Page Template wp:custom page template
  • Put these global template files in theme subfolder page-templates (or any .php files)
  • File name can be anything but follow page_two-columns.php. Don't use page- as prefix!
  • Make a copy of page.php to start
  • Without Template Post Type, global custom page template is only available to post type page
<?php
/**
 * Template Name: Full Width Page
 * Template Post Type: post, page, product
 *
 * @package WordPress
 * @subpackage Twenty_Fourteen
 * @since Twenty Fourteen 1.0
 */
3.23.4.2. archive.php
<?php
/**
 * The template for displaying archive pages
 *
 * @link https://codex.wordpress.org/Template_Hierarchy
 *
 * @package WordPress
 * @subpackage Twenty_Seventeen
 * @since 1.0
 * @version 1.0
 */

get_header(); ?>

<div class="wrap">

    <?php if ( have_posts() ) : ?>
        <header class="page-header">
            <?php
                the_archive_title( '<h1 class="page-title">', '</h1>' );
                the_archive_description( '<div class="taxonomy-description">', '</div>' );
            ?>
        </header><!-- .page-header -->
    <?php endif; ?>

    <div id="primary" class="content-area">
        <main id="main" class="site-main" role="main">

        <?php
        if ( have_posts() ) : ?>
            <?php
            /* Start the Loop */
            while ( have_posts() ) : the_post();

                /*
                 * Include the Post-Format-specific template for the content.
                 * If you want to override this in a child theme, then include a file
                 * called content-___.php (where ___ is the Post Format name) and that will be used instead.
                 */
                get_template_part( 'template-parts/post/content', get_post_format() );

            endwhile;

            the_posts_pagination( array(
                'prev_text' => twentyseventeen_get_svg( array( 'icon' => 'arrow-left' ) ) . '<span class="screen-reader-text">' . __( 'Previous page', 'twentyseventeen' ) . '</span>',
                'next_text' => '<span class="screen-reader-text">' . __( 'Next page', 'twentyseventeen' ) . '</span>' . twentyseventeen_get_svg( array( 'icon' => 'arrow-right' ) ),
                'before_page_number' => '<span class="meta-nav screen-reader-text">' . __( 'Page', 'twentyseventeen' ) . ' </span>',
            ) );

        else :

            get_template_part( 'template-parts/post/content', 'none' );

        endif; ?>

        </main><!-- #main -->
    </div><!-- #primary -->
    <?php get_sidebar(); ?>
</div><!-- .wrap -->

<?php get_footer();

3.23.5. Pagination wp:theme:pagination

https://developer.wordpress.org/themes/functionality/pagination/

the_post_pagination( $args = array() )
echo get_the_posts_pagination( $args )
paginate_links( string|array $args = '' )
old version of the_post_pagination
get_the_posts_pagination()
wp:f:get_the_posts_pagination. Calls paginate_links
posts_nav_link()
next_posts_link()
echo get_next_posts_link()
previous_posts_link()
echo get_previous_posts_link()
<?php if ( have_posts() ) : ?>

    <!-- Add the pagination functions here. -->

    <!-- Start of the main loop. -->
    <?php while ( have_posts() ) : the_post(); ?>

    <!-- the rest of your theme's main loop -->

    <?php endwhile; ?>
    <!-- End of the main loop -->

    <!-- Add the pagination functions here. -->


<div class="nav-previous alignleft"><?php next_posts_link( 'Older posts' ); ?></div>



<div class="nav-next alignright"><?php previous_posts_link( 'Newer posts' ); ?></div>


<?php else : ?>

<?php _e('Sorry, no posts matched your criteria.'); ?>

<?php endif; ?>

Custom Query 

$paged = (get_query_var('paged')) ? get_query_var('paged') : 1;
$args = array(
    'paged' => $paged,
    'orderby' => 'meta_value_num',
    'meta_key' => 'post_views_count',
    'posts_per_page' => 36
);

global $wp_query;
$loop = new WP_Query($args);

if ($loop->have_posts()) :
    while ($loop->have_posts()) : $loop->the_post();
    get_template_part('content', get_post_format());
    endwhile;
    $wp_query = $loop;
    the_posts_navigation();
endif;
wp_reset_query(); // reset main query to original
3.23.5.1. get_the_posts_pagination wp:f:get_the_posts_pagination

get_the_posts_pagination( array $args = array() )

$args
https://developer.wordpress.org/reference/functions/paginate_links/

3.23.6. Setup

wp:f:add_theme_support

3.23.7. Include header, footer, sidebar and other, search form

wp:f:get_header

3.23.8. Widgets

3.23.8.1. Create sidebar

wp:action:widgets_init

3.23.9. Functions used in template

Refer to wp:template tags and wp:t:general

3.23.10. Previous themes

  • Newspaper
  • JobRoller
    • Doc, all articles
    • Post Types
      • job_listing
        • Tax: job_cat, job_type, job_tag, job_salary
      • resume
        • Tax: resume_category, resume_job_type, resume_languages, resume_specialities, resume_groups
      • pricing-plan
      • custom-form

3.24. Genesis Framework wp:genesis

Code snippets (examples)
https://my.studiopress.com/documentation/snippets/
function genesis()
lib/framework.php
(no term)
hooks
Shortcodes
https://genesistutorials.com/genesis-shortcode-list/ search for add_shortcode
Changelog
https://studiopress.github.io/genesis/changelog/

3.24.1. Functions wp:genesis:functions

lib/functions/*.php

3.24.1.1. upgrade.php
3.24.1.2. compat.php
3.24.1.3. general.php
genesis_a11y( $arg = 'screen-reader-text' )
wp:genesis:theme support:genesis-accessibility, return bool
3.24.1.4. options.php wp:genesis:options
header_scripts
string
footer_scripts
string
content_archive
full or excerpts
content_archive_thumbnail
0, not display thumbnail
image_size
'thumbnail', default thumbnail size
breadcrumb_home
0 or 1
breadcrumb_single
breadcrumb_archive
breadcrumb_page
breadcrumb_attachment
(no term)
breadcrumb_404
  1. genesis_get_option( $key, $setting = null, $use_cache = true )
    Known $key
    content_archive
    filter genesis_pre_get_option_$key
    short circuit
    filter genesis_options( get_option( $setting ) or $settings_cache[ $settings ], $setting )
3.24.1.5. image.php
3.24.1.6. markup.php wp:genesis:functions:markup.php
  1. genesis_markup( $args=[] )
    • $args
      echo
      true
      html5
      echo and run genesis_attr( $args['context'] ) if theme support html5 is enabled otherwise xhtml is echoed
      xhtml
      context
      open
      close
      content
    • $pre = apply_filters( "genesis_markup_{$args['context']}", false, $args ); // short circuit
    • $open
      • genesis_attr( $args['context'], array(), $args)
      • filter genesis_markup_$context_open( $open, $args )
    • $content
      • filter genesis_markup_$context_content( $args['content'], $args )
    • $close
      • filter genesis_markup_$context_close( $args['close'], $args )
    • filter genesis_markup_open( $open, $arg )
    • filter genesis_markup_close( $close, $args )
    • echo or return $open.$content.$close
  2. genesis_attr( $context, $attributes = array(), $args = array() ) wp:genesis:functions:genesis_attr
    • genesis_parse_attr() calculates key/value pairs of attribute and value
      • $context becomes class attribute
      • genesis_attr_$context( $attributes, $context, $args )
    • genesis_attr_$context_output( $output, $attributes, $context, $args)
    • filter genesis_attr_$context()
      built in
      • genesis_attr_site-header
      // built in example for site-header
function genesis_attributes_header( $attributes ) {

    $attributes['itemscope'] = true;
    $attributes['itemtype']  = 'https://schema.org/WPHeader';

    return $attributes;

}
      • Then filter genesis_structural_wrap-{$context} is called later
        • genesis_attr_blog-template-description
        • genesis_attr_entry-meta-before-content
        • genesis_attributes_entry()
        • genesis_attr_entry-content
        • genesis_attr_entry-image
        • genesis_attr_entry-image-link
        • genesis_attr_body
        • genesis_attr-nav-$theme_location
        • genesis_attr-breadcrumb
    • filter genesis_attr_$context_output( $output, $attributes, $context, $args )
    1. Sample genesis_attr_body: genesis_attributes_body
      add_filter( 'genesis_attr_body', 'genesis_attributes_body' );
/**
 * Add attributes for body element.
 *
 * @since 2.0.0
 *
 * @param array $attributes Existing attributes for `body` element.
 * @return array Amended attributes for `body` element.
 */
function genesis_attributes_body( $attributes ) {

    $attributes['class']     = implode( ' ', get_body_class() );
    $attributes['itemscope'] = true;
    $attributes['itemtype']  = 'https://schema.org/WebPage';

    // Search results pages.
    if ( is_search() ) {
        $attributes['itemtype'] = 'https://schema.org/SearchResultsPage';
    }

    return $attributes;

}
3.24.1.7. breadcrumb.php
  1. genesis_do_breadcrumbs()
  2. genesis_breadcrumb( $args = [] )

    $args

    prefix
    (no term)
    labels['prefix']
    genesis_markup
    open div context:breadcrumb
    (no term)
    $this->build_crumbs()
    filter genesis_build_crumbs
    $crumbs, $this->args
    (no term)
    suffix
    genesis_markup
    close div context:breadcrumb
    get_archive_crumb()
    filter genesis_archive_crumb
3.24.1.8. menu.php
genesis_nav_menu( $args )
echo genesis_get_nav_menu( $args ); wp:genesis:functions:genesis_nav_menu
(no term)
echo genesis_get_nav_menu( $args )
  • add superfish
  • $nav_markup_open = genesis_structural_wrap( 'menu-' . $sanitized_location, 'open', 0 ); wp:genesis:functions:genesis_structural_wrap
  • $nav = wp_nav_menu( $args )
  • $nav_markup_close = genesis_structural_wrap( 'menu-' . $sanitized_location, 'close', 0 )
  • wrap with <nav></nav> filter genesis_attr-nav-$theme_location, content
  • filter genesis_do_nav( $nav_output, $nav, $args ) or genesis_do_subnav( $nav_output, $nav, $args )
3.24.1.9. layout.php wp:genesis:functions:layout.php
  • genesis_structural_wrap( $context = '', $output = 'open', $echo = true ) wp:genesis:functions:genesis_structural_wrap
    • echo or return a structural wrap div
    • Check if has wp:genesis:theme support:genesis-structural-wraps
    • or </div>">modify attr using genesis_attr( 'structural-wrap' )

    • Finally, filter output genesis_structural_wrap-$context( $output, $original_output ) 

      // no built in example for filter genesis_structural_wrap-$context

// custom code
add_filter( 'genesis_structural_wrap-archive-col1', function($output, $original_output) {
    if ( 'open' == $original_output )  {
        $output = '<div class="wrap archive-wrapper archive-col1">';
    }

    return $output;

}, 10, 2 );
  • genesis_create_initial_layouts()
    • content-sidebar (default)
    • sidebar-content
    • content-sidebar-sidebar
    • sidebar-sidebar-content
    • sidebar-content-sidebar
    • genesis_get_sidebar() will not load the sidebar
    • genesis_register_layout
  • genesis_register_layout( $id = '', $args = array() )
  • genesis_site_layout( $use_cache = true )
    • filter genesis_site_layout to short circuit
    • get layout setting from post, term, genesis_get_cpt_option( 'layout' )
    • If not set above, get it from genesis_get_option( 'site_layout')
      • filter genesis_pre_get_option_site_layout
        • add_filter( 'genesis_pre_get_option_site_layout', '__genesis_return_content_sidebar' );
        • __genesis_return_full_width_content
3.24.1.10. formatting.php
3.24.1.11. seo.php
3.24.1.12. widgetize.php
  • genesis_register_widget_area( $args ) or genesis_register_sidebar
    • Use wp:f:register_sidebar
    • Defined in wp:genesis:action:genesis_setup
    • before_widget, after_widget, before_title, after_title
    • genesis_register_sidebar_defaults genesis_register_wdiget_area_defaults
    • Overide $defaults using $args
    • default registered widget areas
      • header-right
      • sidebar
      • sidebar-alt
      • after-entry
  • wp:f:dynamic_sidebar
    • return false if sidebar isn't found
    • filter genesis_widget_area_defaults, [ before, after, default, show_active, before_sidebar_hook, after_sidebar_hook]
    • action genesis_before_$id_widget_area genesis_after_$id_widget_area
3.24.1.13. feed.php
3.24.1.14. toolbar.php
3.24.1.15. head.php

3.24.2. Structure

3.24.2.1. header.php
3.24.2.2. footer.php
3.24.2.3. menu.php
3.24.2.4. layout.php

Depends on genesis_site_layout in wp:genesis:functions:layout.php

  • load primary sidebar
  • load sidebar-alt
  • add layout name as body_class
3.24.2.5. post.php wp:genesis:structure:post.php
  1. genesis_do_post_format_image()

    Display post format icon

  2. genesis_post_info()

    Display post info (byline)

  3. genesis_do_post_content()
    • is_singular()
      • the_content()
      • filter genesis_edit_post_link
    • 'excerpts' === genesis_get_option( 'content_archive' )
      • the_excerpt()
  4. genesis_reset_loops()
    • add actions
      • genesis_entry_header genesis_do_post_format_image p:4
      • genesis_entry_header genesis_entry_header_markup_open p:5
      • genesis_entry_header genesis_entry_header_markup_close p:15
      • genesis_entry_header genesis_do_post_title
      • genesis_entry_header genesis_post_info p:12
      • genesis_entry_content genesis_do_post_image p:8
      • genesis_entry_content genesis_do_post_content
      • genesis_entry_content genesis_do_post_content_nav p:12
      • genesis_entry_content genesis_do_post_permalink p:14
      • genesis_entry_footer genesis_entry_footer_markup_open p:5
      • genesis_entry_footer genesis_entry_footer_markup_close p:15
      • genesis_after_entry genesis_do_author_box_single p:8
      • genesis_after_entry genesis_adjacent_entry_nav
      • genesis_after_entry genesis_get_comments_template
      • genesis_before_post_title genesis_do_post_format_image
      • genesis_post_title genesis_do_post_title
      • genesis_post_content genesis_do_post_title
      • genesis_post_content genesis_do_post_image
      • genesis_post_content genesis_do_post_content
      • genesis_post_content genesis_do_post_permalink
      • genesis_post_content genesis_do_post_content_nav
      • genesis_before_post_content genesis_post_info
      • genesis_after_post_content genesis_post_meta
      • genesis_after_post genesis_do_author_box_single
      • genesis_loop_else genesis_do_noposts
      • genesis_after_endwhile genesis_posts_nav
    • do action genesis_reset_loops
3.24.2.6. loops.php wp:genesis:loops
  • genesis_do_loop()
  • genesis_standard_loop()
    genesis_before_while
    nothing
    (no term)
    Run while loop the_post();
    genesis_before_entry
    nothing
    genesis_markup
    open <article class="entry">, context:entry
    (no term)
    do action genesis_entry_header
    • bootstrap-for-genesis theme moves gensis_do_post_image from genesis_entry_content to here at p:0
    • genesis_do_post_format_image() at p:4
    • genesis_entry_header_markup_open() at p:5
    (no term)
    <header class="entry-header"> genesis_attr context:entry-header
    • genesis_do_post_title()
    (no term)
    filter genesis_post_title_text( get_the_title() )
    (no term)
    filter genesis_link_post_title( true )
    genesis_markup
    open/close <a class="entry-title-link"> context:entry-title-link
    (no term)
    filter genesis_entry_title_wrap
    genesis_markup
    open/close $wrap <h1 class="entry-title"></h1>, content:~$title~ context:entry-title
    (no term)
    genesis_post_title_output( $output, $wrap, $title)
    genesis_post_info() at p:12
    byline
    (no term)
    filter genesis_post_info $filtered = genesis_post_info( '[post_date]'.__( 'by', 'genesis'). '...' )
    • do_shortcode() p:20
    genesis_markup
    if post type supports genesis-entry-meta-before-content, open/close <p class="entry-meta"> content: genesis_strip_p_tags( $filtered ) context:entry-meta-before-content
    • apply filter genesis_attr_entry-meta-before-content
      genesis_attributes_entry_meta()
      change class to entry-meta
    • close </header>
    genesis_before_entry_content
    nothing
    (no term)
    <div class="entry-content">
    genesis_markup
    context:entry-content
    (no term)
    do action genesis_entry_content
    (no term)
    bootstrap-for-genesis theme moves this to genesis_entry_header
    (no term)
    If not singular and wp:genesis:options has content_archive_thumbnail, display the image
    (no term)
    $img = genesis_get_image()
    genesis_markup
    open/close <a class="entry-image-link"></a>
    (no term)
    close </div> entry-content
    genesis_after_entry_content
    nothing
    (no term)
    genesis_entry_footer
    • <footer class="entry-footer"> genesis_entry_footer_markup_open at p:5
    • Filed under ...
    • </footer> genesis_entry_footer_markup_close at p:15
    genesis_markup
    close </arcitle>, context:entry
    (no term)
    genesis_after_entry
    • genesis_do_author_box_single at p:8
    • genesis_add_id_to_global_exclude at p:9
    • genesis_after_entry_widget_area
    • genesis_adjacent_entry_nav
    • genesis_get_comments_template()
    (no term)
    wp:f:comments_template
    (no term)
    refer to wp:genesis:comments.php
    (no term)
    genesis_after_endwhile
    genesis_posts_nav()
    div.archive-pagination
    • genesis_numeric_posts_nav() or genesis_prev_next_posts_nav()
    genesis_loop_else
    if no post
  • genesis_custom_loop( $args = array() )
    • filter $args = genesis_custom_loop_args
    • $wp_query = new WP_Query( $args )
    • genesis_standard_loop()
    • wp_reset_query()
  • genesis_grid_loop( $args = array() )
    • filter genesis_grid_loop_args
    • remove actions
      • genesis_before_post_title genesis_do_post_format_image
      • genesis_post_content genesis_do_post_image
      • genesis_post_content genesis_do_post_content
      • genesis_post_content genesis_do_post_cotent_nav
      • genesis_entry_header genesis_do_post_format_image
      • genesis_entry_content genesis_do_post_image
      • genesis_entry_content genesis_do_post_content
      • genesis_entry_content genesis_do_post_content_nav
      • genesis_entry_content genesis_do_post_permalink
    • add filter post_class genesis_grid_loop_post_class()
    • add action genesis_post_content genesis_grid_loop_content
    • add action genesis_entry_content genesis_grid_loop_content
    • genesis_standard_loop()
    • genesis_reset_loops()
    • remove filter post_class genesis_grid_loop_post_class()
    • remove action genesis_post_content genesis_grid_loop_content()
    • remove action genesis_entry_content genesis_grid_loop_content()
3.24.2.7. comments.php
3.24.2.8. sidebar.php
3.24.2.9. archive.php
3.24.2.10. search.php

3.24.3. Life cycle

Child theme loads its functions.php before parent genesis theme's functions.php is loaded which is the parent's init.php

Action genesis_pre
nothing
(no term)
Action genesis_init
load textdomain
genesis_i18n()
load theme support
genesis_theme_support() wp:genesis:theme support
load post type support
genesis_post_type_support()
constants
genesis_constants()
load frameworks
genesis_load_framework()
  • do_action( 'genesis_pre_framework' );
  • lib/framework.php
    • function genesis(){}
  • lib/classes/*
  • lib/functions/* wp:genesis:functions
  • lib/shortcodes/* wp:genesis:shortcodes
  • lib/structure/*
  • lib/admin/*
  • lib/js/load-scripts.php
  • lib/css/load-styles.php
  • lib/widgets/*
(no term)
Action genesis_setup wp:genesis:action:genesis_setup
may add_image_size
wp:f:add_image_size
create initial layout
genesis_create_initial_layouts() wp:genesis:functions:layout.php
register default widget areas
genesis_register_default_widget_areas()
3 sidebars are registered
header-right, sidebar, sidebar-alt
(no term)
wp:action:after_setup_theme
  • register 3 default sidebars _genesis_register_default_widget_areas_cb()

3.24.4. Actions and filters

https://genesistutorials.com/visual-hook-guide/

function genesis(){}

  • get_header() wp:genesis:header.php
  • nothing
  • open div, context:content-sidebar-wrap
  • nothing
  • open <main class="content">, context:content
  • genesis_before_loop
    • genesis_do_breadcrumbs()
      • genesis_attr( 'breadcrumb' )
      • choose one function to display. If not found, use genesis_breadcrumb()
        • filter genesis_breadcrumb_args
    • genesis_do_cpt_archive_title_description() wp:genesis:do_cpt_archive_title_description
      • If it's archive page and post type supports archive, then display title for archive page
      • If post type supports genesis-cpt-archives-settings from wp:genesis:post type supports
      • filter genesis_cpt_archive_intro_text_output( $intro_text )
      • do action genesis_archive_title_descriptions( $heading, $intro_text, 'cpt-archive-description' )
        • genesis_do_archive_headings_open() at p:5
      • If $heading or $intro_text, open <div class="cpt-archive-description"> context:cpt-archive-description
        • genesis_do_archive_headings_headline() at p:10
      • ">genesis_attr( 'archive-title' )
        genesis_do_archive_headings_intro_text at p:12
        echo $intro_text
        (no term)
        genesis_do_archive_headings_close at p:15
      • context:cpt-archive-description
    • genesis_do_date_archive_title()
      • If is date archive is_date() then run
      • do action genesis_archive_title_description( $heading, '', 'date-archive-description')
    • genesis_do_blog_template_heading()
      • genesis_attr( 'blog-template-description' )
      • genesis_do_post_title()
        • genesis_archive_title_description
    • genesis_do_posts_page_heading()
    • genesis_do_taxonomy_title_description() at p:15
    • genesis_do_author_title_description() at p:15
    • genesis_do_author_box_archive() at p:15
    • genesis_do_search_title()
  • genesis_loop
    genesis_do_loop
    wp:genesis:loops
    genesis_404
    in template 404.php
  • nothing
  • close </main> context:main
  • genesis_after_content
    • genesis_get_sidebar()
      • get_sidebar()
  • close </div>, context:content-sidebar-wrap
  • genesis_after_content_sidebar_wrap
    • genesis_get_sidebar_alt()
      • get_sidebar( 'alt' )
  • get_footer() wp:genesis:footer.php
3.24.4.1. header.php wp:genesis:header.php

wp:f:get_header loads the parent theme's header.php unless child theme has header.php

  • genesis_doctype
  • genesis_title
  • genesis_meta
  • </head> <body>
    genesis_canonical
    p:5
    (no term)
    genesis_load_favicon
    (no term)
    genesis_do_meta_pingback
    (no term)
    genesis_paged_rel
    (no term)
    genesis_meta_name
    (no term)
    genesis_meta_url
    (no term)
    genesis_header_scripts
    • filter genesis_header_scripts( genesis_get_option( 'header_scripts'))
    (no term)
    genesis_custom_header_style
    (no term)
    remove actions
    • wp_generator
    • adjacent_posts_rel_link_wp_head
    • wlwmanifest_link
    • wp_shortlink_wp_head
    • feed_links_extra
    • rel_canonical
  • open <body class="body">, context:body
  • genesis_before
    • genesis_skip_links() priority 5
  • open div, context:site-container
  • genesis_before_header
    • genesis_skip_links() p:5
  • do action genesis_header
    genesis_header_markup_open() at p:5 context:site-header
    <header class="site-header">
    • genesis_structural_wrap( 'header' ) context:header
      • genesis_attr( 'structural-wrap' )
      • filter genesis_attr_structural-wrap
      • filter genesis_attr_structural-wrap_output
      • filter genesis_structural_wrap-header
    (no term)
    genesis_do_header() at p:10
    genesis_markup
    open, context:title-area
    genesis_site_title
    genesis_seo_site_title()
    genesis_site_description
    genesis_seo_site_description()
    genesis_markup
    close, context:title-area
    genesis_markup~
    open, context:header-widget-area
    genesis_header_right
    nothing
    add filter genesis_header_menu_args
    add filter genesis_header_menu_wrap
    dynamic_sidebar( 'header-right' )
    (no term)
    remove_filters
    genesis_markup
    close, context:header-widget-area
    genesis_header_markup_close at p:15
    close </header>
    • genesis_structural_wrap( 'header', 'close' )
    • context:site-header </header>
    (no term)
    genesis_site_title
    (no term)
    genesis_site_description
    (no term)
    genesis_header_right
    (no term)
    genesis_site_description
  • genesis_after_header
  • open div, context:site-inner
  • genesis_structural_wrap( 'site-inner' )
3.24.4.2. genesis_standard_loop(){}
  • do_action( 'genesis_before_while' );
  • do_action( 'genesis_before_entry' );
  • do_action( 'genesis_entry_header' );
  • do_action( 'genesis_before_entry_content' );
  • do_action( 'genesis_entry_content' );
  • do_action( 'genesis_after_entry_content' );
  • do_action( 'genesis_entry_footer' );
  • do_action( 'genesis_after_entry' );
  • do_action( 'genesis_after_endwhile' );
3.24.4.3. footer.php wp:genesis:footer.php

wp:f:get_footer loads parent theme's footer.php unless child theme has footer.php

  • genesis_structural_wrap( 'site-inner', 'close')
  • close </div>, context:site-inner
  • genesis_before_footer
    • genesis_footer_widget_areas()
      • dynamic_sidebar( 'footer-'. $counter )
      • open/close <div class="footer-widget-area">, context:footer-widget-area, content:$widgets
      • genesis_structural_wrap( 'footer-widgets', 'open', 0)
      • genesis_structural_wrap( 'footer-widgets', 'close', 0)
      • open/close <div class="footer-widgets">, context:footer-widgets
  • genesis_footer
    • genesis_footer_markup_open p:5
      genesis_markup
      open <footer class="site-footer">, context:site-footer
      (no term)
      genesis_structurual_wrap( 'footer', 'open' )
    • genesis_do_footer
      filter genesis_footer_backtotop_text
      nothing
      filter genesis_footer_creds_text
      $creds_text nothing
      (no term)
      filter genesis_footer_output
      • :: do_shortcode() p:20
    • genesis_footer_markup_close p:15
      • genesis_structural_wrap( 'footer', 'close' )
      • close </footer>, context:site-footer
  • nothing
  • close </div>, context:site-container
  • nothing
  • wp_footer()
  • close </body>, context:body
3.24.4.4. comments.php wp:genesis:comments.php
genesis_before_comments
nothing
(no term)
genesis_comments
  • genesis_do_comments()
    genesis_markup
    open <div class="entry-comments">, context:entry-comments
    (no term)
    filter genesis_title_comments
    (no term)
    actiion genesis_list_comments
    • genesis_default_list_comments()
    (no term)
    filter genesis_comment_list_args
    (no term)
    wp_list_comments( $args )
    (no term)
    prev link genesis_prev_comments_link_text
    (no term)
    next link genesis_next_comments_link_text
    genesis_markup
    open/close div, content:$pagination, context:comments-pagination
    genesis_markup
    close div, context:entry-comments
genesis_after_comments
none
genesis_before_pings
none
(no term)
genesis_pings
  • genesis_do_pings()
    • filter genesis_attr_entry-pings
    • open div, context:entry-pings
    • genesis_list_pings
    • close div, context:entry-pings
genesis_after_pings
none
genesis_before_comment_form
none
(no term)
genesis_comment_form
genesis_do_comment_form()
genesis_after_comment_form
none

3.24.5. Theme support wp:genesis:theme support

3.24.6. Post Type Supports wp:genesis:post type supports

Refer to wp:f:add_post_type_support

genesis-seo
enable seo fields on wp-admin
genesis-scripts
add a field for per-post script on wp-admin
genesis-layouts
default layout for a CPT
genesis-cpt-archives-settings
enable archive settings
genesis-entry-meta-before-content
disable genesis_post_info (byline)
genesis-entry-meta-after-content
disable <footer></footer> and genesis_post_meta()
genesis-after-entry-widget-area
disable <div class="after-entry widget-area"></div>
(no term)
genesis-adjacent-entry-nav

3.24.7. Shortcodes wp:genesis:shortcodes

All shortcodes have filters

  • [footer_home_link after=""]
  • [footer_loginout after="" before="" redirect=""]
  • [footer_backtotop before="" after="" href="#wrap" nofollow="true" text="return to top of page"]
    • If genesis_html5(), then empty
  • [footer_copyright before="" after="" copyright="&#x000A9;"]
    • sprintf( '[footer_copyright before="%s "] &#x000B7; [footer_loginout]', __( 'Copyright', 'genesis' ) );
  • [footer_home_link before="" after="" text="get_bloginfo( 'name' )"]

3.24.8. Child theme

functions.php 

add_action( 'genesis_setup', 'bfg_childtheme_setup', 15 );

function bfg_childtheme_setup() {
    // Load parent Genesis theme init.php
    include_once( get_template_directory() . '/lib/init.php' );
    // ...
}

3.25. Customize API wp:api:customize

3.25.1. WP_Customize_Manager wp:WP_Customize_Manager

  • Panels > Sections > Controls > Settings
  • wp:action:customize_register">do_action( 'customize_register', $wp_customize );

    • add_setting, add_control, add_section, add_panel Refer to wp:f:get_theme_mod after add_setting 

      add_action( 'customize_register', function ( $wp_customize ) {
    $wp_customize->add_setting( 'my_setting', array(
        'default' => '', // Still need to save/Publish value on Appearance > My Section > My Control > My Setting
        // 'type' => 'theme_mod', // default. Use get_theme_mod later
    ) );

    $wp_customize->add_section( 'my_section', array(
        'title'    => __( 'My Section', 'mytheme' ),
        'priority' => 30, // d:160
        // 'panel' => '', // default
    ) );

    $wp_customize->add_control( new WP_Customize_Control(
        $wp_customize,
        'my_control_for_my_setting', // or simply use my_setting
        array(
            'label'    => __( 'My Control Title', 'theme_name' ),
            'section'  => 'my_section',
            'settings' => 'my_setting',
            'type'     => 'text',
        )
    ) );

} );

3.26. Admin

3.26.1. Toolbar - Admin bar wp:admin:toolbar

  • Admin bar is replaced with Toolbar since WP 3.3
  • wp_footer(); is called. <body> will have class admin-bar and /wp-includes/css/admin-bar.min.css is added
  • show_admin_bar( false ); is not run in functions.php
  • In newer version of wp, wp:filter:show_admin_bar can be used to turn off the admin bar

  • If you have sticky or position:fixed element .sticky-header 

    .sticky-header {
  position: fixed;
  top: 0;
}
.admin-bar .sticky-header {
  top: 46px; /* 0 + 46 */
}
@media screen and (min-width: 783px) {
  .admin-bar .sticky-header {
top: 32px; /* 0 + 32 */
  }
}

3.26.2. List Table API WP_Posts_Lists_Table wp:api:list table

  • wp-admin/class-wp-posts-list-table.php
3.26.2.1. Action manage_${post_type}_posts_custom_column wp:api:list table:action:manage_postspost_type_custom_column
  • do_action( "manage_{$post->post_type}_posts_custom_column", $column_name, $post->ID );
  • echo column values
  • For native post types
    • manage_posts_custom_column
    • manage_pages_custom_column
    • No!!! manage_users_custom_column is a filter not an action!
    • manage_comments_custom_column
    • manage_media_custom_column
    • manage_plugins_custom_column
    • manage_themes_custom_column
    • manage_link_custom_column
    • manage_sites_custom_column
  • Populate or modify column values
add_action( 'manage_book_posts_custom_column', function ( $column, $post_id ) {
    switch ( $column ) {

        case 'book_author' :
            $terms = get_the_term_list( $post_id, 'book_author', '', ',', '' );
            if ( is_string( $terms ) ) {
                echo $terms;
            } else {
                _e( 'Unable to get author(s)', 'your_text_domain' );
            }
            break;

        case 'publisher' :
            echo get_post_meta( $post_id, 'publisher', true );
            // custom post meta
            // echo get_post_meta( $post_id, '_custom_post_order', true)
            // ACF plugin
            // echo get_field('custom_order', $post_id);
            break;

    }
}, 10, 2 );
3.26.2.2. Filter manage_${taxonomy}_custom_column and manage_users_custom_column wp:api:list table:filter:managetaxonomy_custom_column
3.26.2.3. Filter manage_${post_type}_posts_columns wp:api:list table:filter:managepost_type_posts_columns
  • Add/remove columns
  • Almost the same as wp:api:plugin:filter:managescreen_id_columns, but this one runs after
add_filter( 'manage_book_posts_columns', function ( $columns ) {
    // var_dump('CPT book manage columns filter'); var_dump($columns);
    // unset( $columns['author'] ); //  don't show a column

    $columns['book_author'] = __( 'Author', 'your_text_domain' ); // add a column
    $columns['publisher']   = __( 'Publisher', 'your_text_domain' ); // add another column

    return $columns;
} );
3.26.2.4. Filter manage_{screen_id}_columns wp:api:list table:filter:managescreen_id_columns
  • e.g. apply_filters( 'manage_posts_columns', $posts_columns, $post_type );
  • Almost the same as wp:api:plugin:filter:managepost_type_posts_columns
  • screen_id
    • edit-my_tax_name
    • refer to wp:api:plugin:action:manage_postspost_type_custom_column
3.26.2.5. Filter manage_{screen_id}_sortable_columns wp:api:list table:filter:managescreen_id_sortable_columns
  • apply_filters( "manage_{$this->screen->id}_sortable_columns", $sortable_columns );
  • Refer to wp:api:taxonomy
  • screen_id
    • edit-my_tax_name
    • refer to wp:api:plugin:action:manage_postspost_type_custom_column

3.27. Custom Cache

Same page request custom cache, Single request cache 

function lili_request_cache($field, $set = null) {
    static $custom_cache = [];
    if ($set !== null) {
        $custom_cache[$field] = $set;
    }
    elseif ( ! isset($custom_cache[$field]) || $custom_cache[$field] == null) {
        // Set default by field
        switch ($field) {
            case 'c':
                $custom_cache[$field] = 'default value for c';
                break;
            default:
                $custom_cache[$field] = null;
                break;
        }
    }

    return $custom_cache[$field];
}

lili_request_cache('a', 1);
var_dump('a: ', lili_request_cache('a')); // 1
var_dump('b: ', lili_request_cache('b')); // null
var_dump('a: ', lili_request_cache('a')); // 1
lili_request_cache('a', 2);
var_dump('a: ', lili_request_cache('a')); // 2
var_dump('c: ', lili_request_cache('c')); // default value for c

3.28. Custom page

wp:filter:template_include

3.29. Database wp:db

Database Description

3.29.1. wp_posts wp:db:wp_posts

ID
bigint(20)
post_author
bigint(20)
post_date
datetime
post_date_gmt
datetime. Refer to wp:post:post_date_gmt
post_content
longtext
post_title
text
post_excerpt
text
post_status
varchar(20) d:'publish' Refer to wp:f:register_post_status
post_name
varchar(200) d:'' name in slug
to_ping
text
ping_status
varchar(20) d:'open'
pinged
text
post_modified
datetime
post_modified_gmt
datetime
post_content_filtered
longtext
post_parent
bigint(20)
guid
varchar(255)
menu_order
int(11)
post_type
varchar(20) d:'post'
Core
post, page, attachment, nav_menu_item, revision, oembed_cache wp:api:oembed
(no term)
Custom post type slug
post_mime_type
vchar 100 blank for non media posts. e.g. image/jpeg
comment_status
varchar(20) d:'open'
comment_count
bigint(20)
post_password
varchar(20) d:''

#+NAME Number of posts per Post Type, Year, Month 

SELECT
  -- YEAR(p.post_date)  AS year,
  -- MONTH(p.post_date) AS month,
     post_type,
     count(*),
     -- ,p.ID, p.post_title, p.post_type, p.post_status
     1
FROM wp_posts p
WHERE 0 = 0
  AND p.post_type NOT IN ('attachment', 'revision', 'nav_menu_item', 'oembed_cache')
  -- AND p.post_status = 'publish'
GROUP BY
  -- YEAR(p.post_date),
  -- MONTH(p.post_date),
  post_type
ORDER BY year DESC, month DESC,
     post_date DESC
-- LIMIT 100

Get terms for posts 

SELECT
   t.term_id, t.name as term_name, t.slug as term_slug, t.term_group,
   tr.term_order,
   tt.term_taxonomy_id, tt.taxonomy as taxonomy, tt.parent,
   p.ID as post_id, p.post_title, p.post_type, p.post_status,
    -- count(*),
   1
FROM wp_posts p
     INNER JOIN wp_term_relationships tr ON tr.object_id = p.ID
     INNER JOIN wp_terms t ON t.term_id = tr.term_taxonomy_id
     INNER JOIN wp_term_taxonomy tt ON tt.term_id = t.term_id
WHERE 1 = 1
  AND p.post_type NOT IN ('attachment', 'revision', 'nav_menu_item', 'oembed_cache', 'wp-types-group')
  AND p.post_status = 'publish'
  -- AND tt.taxonomy = 'post_tag'
  -- AND ( t.slug LIKE 'TTNA' OR t.name LIKE 'TTNA' )
  -- AND t.slug LIKE 'GCKC'
  -- AND p.ID = '144571'
-- GROUP BY t.term_id,
     -- tt.taxonomy,
     -- 1
ORDER BY p.post_date DESC

3.29.2. wp_terms

term_id
int, auto_increment
name
varchar(200)
slug
varchar(200)
term_group
int

3.29.3. wp_term_relationships wp:db:wp_term_relationships

object_id
int, (e.g. ID in wp_posts), 0
term_taxonomy_id
int, 0
term_order
int, 0

#+NAME Return posts that don't have certain terms 

SELECT p.ID as post_id,
   p.post_title,
   p.post_type,
   p.post_status,
   CONCAT('https://www.a.ca/wp-admin/post.php?action=edit&post=' , p.ID)
FROM wp_posts p
WHERE 1 = 1
  AND p.post_type IN ('vendors')
  -- AND p.post_status IN ('publish', 'draft')
  AND NOT EXISTS(
    SELECT *
    FROM wp_term_relationships tr
         INNER JOIN wp_terms t ON t.term_id = tr.term_taxonomy_id
         INNER JOIN wp_term_taxonomy tt ON tt.term_id = t.term_id
    WHERE tr.object_id = p.ID
      AND tt.taxonomy = 'vendor_city'
      AND t.slug IN ('slug1', 'slug2')
    )
ORDER BY p.post_date DESC

3.29.4. wp_term_taxonomy wp:db:wp_term_taxonomy

term_taxonomy_id
bigint(20)
term_id
bigint(20)
taxonomy
varchar(20) category, post_tag, link_category, nav_menu, ngg_tag
description
longtext
parent
bigint(20)
count
bigint(20) wp:db:wp_term_taxonomy:count

3.29.5. wp_termmeta

meta_id
bigint(20) unsigned, PK, start from 1
term_id
bigint(20) unsigned, IND, default 0
meta_key
varchar(255) e.g. wpcf-publication_date
meta_value
longtext

3.29.6. wp_postmeta wp:db:wp_postmeta

meta_id PK
bigint 20
post_id
bigint 20
meta_key
vchar(255) if custom field has prefix _, It means the field is private and does not display on Post Edit page on wp-admin
  • wp:plugin:acf key custom_order with value from wp-admin and _custom_order with value field_ALPHANUMERICRANDOM which refers table wp_posts with post_name = 'field_ALPHANUMERICRANDOM'

  • wp:custom page template e.g. page-news.php default 

    SELECT
    pm.meta_value, COUNT(*)
   -- ,p.id, p.post_title, p.post_type, p.post_parent, p.post_date, p.guid, p.post_name
FROM wp_posts p
     INNER JOIN wp_postmeta pm ON pm.post_id = p.id
WHERE 1=1
  AND p.post_status = 'publish'
  AND pm.meta_key = '_wp_page_template'
  -- AND p.ID = '126126'
-- AND pm.meta_value = 'page-no-title.php'
GROUP BY pm.meta_value
ORDER BY p.post_date DESC
  • unix_timestamp:userid e.g. 1566567414:38
  • userid e.g. 38
meta_value
longtext string or php serialized string

#+NAME Get all meta keys and values for a post 

SELECT p.post_type, p.ID, pm.meta_key, pm.meta_value, pm.meta_id
FROM wp_posts p
     INNER JOIN wp_postmeta pm ON pm.post_id = p.id
WHERE 1=1
  AND p.ID = '123'

#+NAME Get all posts with a meta key 

SELECT p.post_type, p.ID, pm.meta_key, pm.meta_value, pm.meta_id
-- ,CAST(pm.meta_value AS Date)
FROM wp_posts p
   INNER JOIN wp_postmeta pm ON pm.post_id = p.id
WHERE 1=1
--  AND p.post_type = 'agenda'
  AND p.post_status = 'publish'
  AND pm.meta_key LIKE '%agenda_date%'
ORDER BY p.post_date DESC
3.29.6.1. meta_key Media files
_wp_attached_file
2009/11/a.jpg 2014/02/a.pdf combine with wp:db:wp_options:upload_path wp:db:wp_postmeta:_wp_attached_file
_wp_attachment_image_alt
string
_thumbnail_id
int, this is the featured image post id (123) of parent post (122)
  • Featured image post 123 with post_parent = the real parent post (122)
  • Featured image post 123 also has these meta keys and values described here
  • Featured image post 123 has post_type as attachment
(no term)

_wp_attachement_backup_sizes 

Array
(
[full-orig] => Array
    (
    [width] => 1024
    [height] => 768
    [file] => Tree.jpg
    )

[full-1317235120383] => Array
    (
    [width] => 316
    [height] => 237
    [file] => Tree-e1317235110739.jpg
    )

)
_wp_attachment_metadata

serialized string. Only images have non-empty array. It contains all wp:f:add_image_size for a featured image 

Array
(
[width] => 100
[height] => 100
[hwstring_small] => height='96' width='96'
[file] => 2009/11/competitiveedge_091109_tease.jpg
[sizes] => Array
    (
    [thumbnail] => Array
        (
        [file] => competitiveedge_091109_tease-100x60.jpg
        [width] => 100
        [height] => 60
        )

    )

[image_meta] => Array
    (
    [aperture] => 0
    [credit] => 
    [camera] => 
    [caption] => 
    [created_timestamp] => 0
    [copyright] => 
    [focal_length] => 0
    [iso] => 0
    [shutter_speed] => 0
    [title] => 
    )

)

3.29.7. wp_options wp:db:wp_options

3.30. Plugins

3.30.2. GitHub Updater wp:plugin:github-updater

Update and install GitHub, Bitbucket, GitLab and Gitea hosted plugins and themes
https://github.com/afragen/github-updater
Installation
just download the zip

3.30.3. Classic Editor wp:plugin:classic-editor

3.30.4. Development

3.30.4.1. Simply Show Hooks wp:plugin:simply-show-hooks

3.30.5. Translation

3.30.5.1. WPML wp:plugin:sitepress-multilingual-cms wp:plugin:wpml
  • Before there's lifetime license but now only yearly
  • 3 levels of license: Multilingual Blog, CMS and Agency. Agency can have unlimited website registration which provides plugin update notification
  • Working with themes and ohter plugins
  • Change language flag Languages > Site Languages > Edit languages
    pluginfolder/res/flags
    size 18x12
  • Settings > Taxonomy Translation
    • Translatable - only show translated items (when you edit a single post)
  • Taxonomy translation
    copy to all languages
    copy default language setting to other languages
    (no term)
    Refresh permalinks after
  1. Translate Widgets

    Requires String Translation addon. Change domain to Widgets.

  2. Template
    Get languages
    https://wpml.org/wpml-hook/wpml_active_languages/ 
    $languages = apply_filters( 'wpml_active_languages', NULL, 'orderby=id&order=desc' );
    Homepage URL
    <a href="<?php echo apply_filters( 'wpml_home_url', get_option( 'home' ) ); ?>">Home</a>
  3. Manual update plugin
    • Deactivate it and other addons
    • Copy folder
    • Activate
  4. shortcodes
    1. wpml-string

      Requires WPML String Translation [wpml-string context="my-domain" name="my-name"]My string[/wpml-string] To translate this string, go to the WPML -> String Translation page and use the following info (replacing the names with your own):

      • Domain: my-domain
      • Name: my-name
      • String: My string

      Pay attention to the page when the string is registered. If the string is first registered in French, then you should Change the language of selected strings to French.

3.30.5.2. polylang wp:plugin:polylang

https://polylang.wordpress.com/documentation/

Manage multilingual posts in one post per language (e.g. WPML - paid, xili-language, Polylang, Bogo or Sublanguage). Translations are then linked together, indicating that one page is the translation of another.

There are other types of multilingual plugins:

  • Store all languages alternatives for each post in the same post (e.g. qTranslate-X, WPGlobus).
  • Manage translations on the generated page instead of using a post context (e.g. Transposh and Global Translator).
  • Plugins like Multisite Language Switcher, Multilingual Press, and Zanto, link together separate WordPress network (multisite) installations for each language by pinging back and forth.

Add languages including English (the first language) Click on a post/page, click on + for another language and then create the translation page. You can select another existing page and add that page to the current page's translation. Posts in different languages will be created with different URLs. Go to edit one post and you can go to different translated pages.

URLs 

/sample-page
/zh/chinese-sample-page

You can create a different menu for each language

Determine the locale of current page 

//get_locale() :: 'en_US'
//get_bloginfo('language') :: 'en-US'

// in template
<div class="nav-previous alignleft">
<?php
  $currentlang = get_bloginfo('language');
  if($currentlang=="en-US"):
?>
This is English
<?php else: ?>
This is Spanish
<?php endif; ?>

</div>

Display links to posts translations within the loop 

<?php while ( have_posts() ) : the_post(); ?>
<ul class='translations'><?php pll_the_languages( array( 'post_id' => $post->ID ) ); ?></ul>
<?php the_content(); ?>
<?php endwhile; ?>

Other functions https://polylang.wordpress.com/documentation/documentation-for-developers/functions-reference/ 

<?php // outputs a list of languages names ?>
<ul><?php pll_the_languages(); ?></ul>

<?php // outputs a flags list (without languages names) ?>
<ul><?php pll_the_languages(array('show_flags'=>1,'show_names'=>0)); ?></ul>

<?php // outputs a dropdown list of languages names ?>
<?php pll_the_languages(array('dropdown'=>1)); ?>
  1. String translation is possible

    There're 3 ways to register a string to translate

    1. pll_register_string($name, $string, $group, $multiline);
      • $name => (required) name provided for sorting convenience (ex: ‘myplugin’)
      • $string => (required) the string to translate
      • $group => (optional) the group in which the string is registered, default is ‘polylang’
      • $multiline => (optional) if set to true, the translation text field will be multiline, default is false
      function register_strings() {
  pll_register_string('mytheme', 'some_string');
}
add_action('init','register_strings');

// in template to echo
pll_e('some_string');

// to return
pll__('some_string');

// return in another language that might not be the current language
pll_translate_string($string, $lang);
    2. WPML API

      for example icl_register_string($context, $name, $string);

    3. wpml-config.xml

      Some plugin or theme might include this WPML plugin xml in its root directory. https://polylang.pro/doc/the-wpml-config-xml-file/ 

      <wpml-config>
    <custom-fields>
    <custom-field action="copy">quantity</custom-field>
    <custom-field action="translate">custom-title</custom-field>
    </custom-fields>
    <custom-types>
    <custom-type translate="1">book</custom-type>
    <custom-type translate="1">DVD</custom-type>
    </custom-types>
    <taxonomies>
    <taxonomy translate="1">genre</taxonomy>
    </taxonomies>
    <admin-texts>
    <key name="my_plugins_options">
        <key name="option_name_1" />
        <key name="option_name_2" />
        <key name="options_group_1">
        <key name="sub_option_name_11" />
        <key name="sub_option_name_12" />
        </key>
    </key>
    <key name="simple_string_option" />
    </admin-texts>
</wpml-config>

      custom-fields: the custom field name needs to be provided and also the action

      ignore
      no action
      translate
      value is copied from the source but may be modified
      copy
      value is copied from the source and synchronized across translations

      custom-types: the custom post types that Polylang should manage. Once translate attribute is set, it cannot be changed in Polylang settings. 1 is to manage, 0 is not to manage.

      taxonomies: the custom tax that Polylang should manage.

      admin-texts: when get_option is called in themes or plugins, Polylang can filter these calls and provide translation to the values of these options.

      • Add a key name. A key can have children for serialization.
  2. Sitemap and Permalink
    • After adding new language, the sitemap will be empty including the default language
    • wp:plugin:all-in-one-seo-pack has to have Polylang set to "The language is set from different domains"
      • originaldomain.com/page1
      • fr.originaldomain.com/page1-french-name
    • Even the domain is different, the permalinks of the 2 pages in 2 different languages have to be different.
      • originaldomain.com/sitemap.xml
      • fr.originaldomain.com/sitemap.xml

3.30.6. Image

3.30.6.1. Resize image on the fly fly-dynamic-image-resizer
$image = fly_get_attachment_image_src(
 get_post_thumbnail_id(),
 array(160, 9999),
 false // true to crop to exact dimension
);

$image :: array(
 'src',
 'width',
 'height'
);

fly_add_image_size('lili-w600', 600, 9999, true);
$image = fly_get_attachment_image($post->id, 'lili-w600'); // HTML
3.30.6.2. Imsanity - Resize Image wp:plugin:imsanity
  • I think gif will not be included which is good
  • Resize upload images
  • If Convert PNG to JPG is No, then PNG file will not be resized automatically
  • Batch processing existing images
    • Some image files may fail. Don't select them
    • May use global variable to change the default 250 images returned for batch processing
3.30.6.3. Crop-Thumbnails wp:plugin:crop-thumbnails
  • Resize featured image, Crop featured image, crop thumbnail
3.30.6.4. multiple-featured-images wp:plugin:multiple-featured-images
add_filter( 'kdmfi_featured_images', function( $featured_images ) {
    $args_1 = array(
    'id' => 'featured-image-2',
    'desc' => 'Your description here.',
    'label_name' => 'Featured Image 2',
    'label_set' => 'Set featured image 2',
    'label_remove' => 'Remove featured image 2',
    'label_use' => 'Set featured image 2',
    'post_type' => array( 'page' ),
    );

    $args_2 = array(
    'id' => 'featured-image-3',
    'desc' => 'Your description here.',
    'label_name' => 'Featured Image 3',
    'label_set' => 'Set featured image 3',
    'label_remove' => 'Remove featured image 3',
    'label_use' => 'Set featured image 3',
    'post_type' => array( 'page', 'post' ),
    );

    $featured_images[] = $args_1;
    $featured_images[] = $args_2;

    return $featured_images;
});

3.30.7. Image, Media in cloud

3.30.7.1. S3-Uploads wp:plugin:S3-Uploads
  • https://github.com/humanmade/S3-Uploads
  • composer require humanmade/s3-uploads or download and extract manual-install.zip

  • wp-config.php 

    define( 'S3_UPLOADS_BUCKET', 'media.a.com' );
define( 'S3_UPLOADS_KEY', 'xxx' );
define( 'S3_UPLOADS_SECRET', 'yyy' );
define( 'S3_UPLOADS_REGION', 'us-east-1' );
define( 'S3_UPLOADS_BUCKET_URL', 'https://media.a.com' ); // without trailing slash. If not defined, file urls in db will start with [bucketname].s3.amazonaws.com/uploads/[file path]
define( 'S3_UPLOADS_AUTOENABLE', false ); // True to redirect all existing media files in db to `s3://`. Before doing that, upload all existing files to S3 first.
  • wp plugin activate S3-Uploads
  • wp s3-uploads verify
  • wp s3-uploads create-iam-user --admin-key=<key> --admin-secret=<secret>
    If you want to create IAM user yourself, use this policy
    wp s3-uploads generate-iam-policy
  • wp s3-uploads ls [<path>]
  • wp s3-uploads upload-directory wp-content/uploads/ uploads
    • May use --sync and --dry-run
  • wp s3-uploads cp ./text.txt s3://mybucket/test.txt
  • use aws:cloudfront to avoid setting the following
    Expire in 30 days
    define( 'S3_UPLOADS_HTTP_CACHE_CONTROL', 30 * 24 * 60 * 60 );
    On top, set Expires header e.g. to set an asset to not expire
    define( 'S3_UPLOADS_HTTP_EXPIRES', gmdate( 'D, d M Y H:i:s', time() + (10 * 365 * 24 * 60 * 60) ) .' GMT' ); e.g. expire in 10 years
  • define( 'S3_UPLOADS_DISABLE_REPLACE_UPLOAD_URL', true );
  • define('S3_UPLOADS_OBJECT_ACL', 'private');
  • Different endpoints e.g. Digital Ocean Spaces
  • Local dev
    • Disable the plugin or
    • mock S3 with a local stream wrapper and actually store the uploads in wp-content/uploads/s3/
3.30.7.2. WP Offload Media (formerly WP Offload S3) wp:plugin:amazon-s3-and-cloudfront
  • Free version
    • Cloud storage providers supported: S3, DigitalOcean Spaces, Google Cloud Storage
    • Only newly uploaded files will be copied to the cloud
  • Pro version: https://deliciousbrains.com/wp-offload-media/
    • Plans
      • Bronze: 2k offloaded media items across unlimited sites
      • Silver: 6k
      • Gold: 20k, multisite, all integrations with addons $199/yr
      • Platinum: 40k, and Gold
      • Adamantium: 100k, and Platinum

    • Offloaded media item is 1 item in Media Library with all resized images

      • If exceeded, these will be disabled
      • Offload of existing media library tool
      • Download to server tool
      • Remove from server tool
      • On demand bucket actions for attachments (Copy to Bucket or Remove from Server in Media page)
      • WooCommerce integration for new product files
      • New files set to private for WooCommerce
      • New files set to private for wp:plugin:easy-digital-downloads
        • treat the Plan difference as the initial upload size of WP website
    • Upload existing Media Library
    • Assets Pull Addon: serve CSS, JS, images, fonts from CloudFront. Available with Gold license
    • wp:plugin:acf, wp:plugin:woocommerce, wp:plugin:wpml, wp:plugin:easy-digital-downloads, wp:plugin:metaslider

3.30.8. User, Roles, Capability

3.30.8.1. User Role Editor wp:plugin:user-role-editor
  • Add or remove capapbilities to a role
  • Add or remove capabilities to a user
3.30.8.2. Capability Manager Enhanced wp:plugin:capability-manager-enhanced
  • Create Roles and assign capabilities

3.30.9. Fields

3.30.9.1. ACF wp:plugin:advanced-custom-fields wp:plugin:acf

WP.org

  • Add a field group and then add custom fields under the group.
  • Use Location to define rules to assign the field group to a custom post type
  • Hide the default post fields (Content editor, comments, categories, etc.)
  • Make custom fields required
  • wp:The_Loop, to display a custom field">the_field('field_name')
  • get_field('field_name');
  1. Image field

    3 types of return

    • array
    • URL
    • ID

    https://www.advancedcustomfields.com/resources/image/

  2. Post object field
    echo '<pre>';
    print_r( get_field('my_post_objects')  );
echo '</pre>';

// single post object
$post_object = get_field('my_post_object');
global $post;
if( $post_object ): 
    // override $post
    $post = $post_object;
    setup_postdata( $post ); 
    ?>
    <div>
    <h3><a href="<?php the_permalink(); ?>"><?php the_title(); ?></a></h3>
    <span>Post Object Custom Field: <?php the_field('field_name'); ?></span>
    </div>
    <?php wp_reset_postdata(); // IMPORTANT - reset the $post object so the rest of the page works correctly ?>
<?php endif; ?>

// multiple post objects
$post_objects = get_field('my_post_objects');

if( $post_objects ): ?>
    <ul>
    <?php foreach( $post_objects as $post_object): ?>
    <li>
        <a href="<?php echo get_permalink($post_object->ID); ?>"><?php echo get_the_title($post_object->ID); ?></a>
        <span>Post Object Custom Field: <?php the_field('field_name', $post_object->ID); ?></span>
    </li>
    <?php endforeach; ?>
    </ul>
<?php endif;
  3. Repeater field
    <?php if ( have_rows( 'repeater_field_name' ) ): ?>

    <ul class="slides">

        <?php while ( have_rows( 'repeater_field_name' ) ): the_row();

            // vars
            $image    = get_sub_field( 'image' );
            $content  = get_sub_field( 'content' );
            $link     = get_sub_field( 'link' );
            $post_obj = get_sub_field( 'my_post_object' );
            ?>

        <li class="slide">

                <?php if ( $link ): ?>
        <a href="<?php echo $link; ?>">
                    <?php endif; ?>

            <img src="<?php echo $image['url']; ?>" alt="<?php echo $image['alt'] ?>"/>

                    <?php if ( $link ): ?>
        </a>
            <?php endif; ?>

                <?php echo $content; ?>
        <?php echo get_the_title($post_obj->ID); ?>

        </li>

        <?php endwhile; ?>

    </ul>

<?php endif; ?>

    Get first row or random row 

    $rows = get_field('repeater_field_name' ); // get all the rows
$first_row = $rows[0]; // first row
$rand_row = $rows[ array_rand( $rows ) ]; // get a random row
$rand_row_image = $rand_row['sub_field_name' ]; // get the sub field value 

// Note
// $first_row_image = 123 (image ID)

$image = wp_get_attachment_image_src( $rand_row_image, 'full' );
// url = $image[0];
// width = $image[1];
// height = $image[2];
?>
<img src="<?php echo $image[0]; ?>" />
  4. Relationship field
    <?php

// speaker post type has a relationship field speaker_agenda to relate to post type agenda
// the following example's main query is single-speaker
$posts = get_field( 'relationship_field_name' ); // returns an array of WP_Post obj

if ( $posts ): ?>
    <ul>
        <?php
        // global $post; // Required if foreach uses $post as item
        foreach ( $posts as $p ): // variable must NOT be called $post (IMPORTANT)
            ?>
        <li>
        <a href="<?php echo get_permalink( $p->ID ); ?>"><?php echo get_the_title( $p->ID ); ?></a>
        <span>Custom field from $post: <?php the_field( 'author', $p->ID ); ?></span>
        </li>
        <?php endforeach; ?>
    </ul>
<?php endif;

// return an array of post ids
$session_ids = get_field( 'speaker_agenda', false, false ); // first false is to use the global $post, 2nd false is not to format the value as a post obj

// further sort
$query = new WP_Query( array(
    'post_type'      => 'agenda',
    'posts_per_page' => - 1,
    'post__in'       => $session_ids,
    'post_status'    => 'publish',
    'orderby'        => 'post__in',
) );

// reverse query. The following's main query is single agenda
$speakers = get_posts( array(
    'post_type'  => 'speaker',
    'meta_query' => array(
        array(
            'key'     => 'relationship_field_name',
            // name of custom relationship field
            'value'   => '"' . get_the_ID() . '"',
            // matches exactly "123", not just 123. This prevents a match for "1234"
            'compare' => 'LIKE'
        )
    )
) );
if ( $speakers ): ?>
    <ul>
        <?php foreach ( $speakers as $speaker ): ?>
            <?php
            $photo = get_field( 'photo', $speaker->ID );
            ?>
        <li>
        <a href="<?php echo get_permalink( $speaker->ID ); ?>">
            <img src="<?php echo $photo['url']; ?>" alt="<?php echo $photo['alt']; ?>" width="30"/>
                    <?php echo get_the_title( $speaker->ID ); ?>
        </a>
        </li>
        <?php endforeach; ?>
    </ul>
<?php endif; ?>
  5. Query
    if ( in_array( $query->get( 'post_type' ), array( 'event' ) ) ) {

    $query->set( 'orderby', 'meta_value' );
    $query->set( 'meta_key', 'event_start' );
    $query->set( 'order', 'ASC' );
    $query->set( 'posts_per_page', '-1' );

    $now        = strtotime( '12:00:00' );
    $meta_query = array(
        'relation' => 'AND',
        array(
            'key'     => 'event_end',
            'value'   => date( 'Ymd'),
            // value in db is '20190131' for Jan 31, 2019. So the format should Ymd
            //'value'   => date( 'Ymd', strtotime( '-12 day', $now ) ),
            'compare' => '>=',
        )
    );
    $query->set( 'meta_query', $meta_query );
}
  6. Add a field group

    https://www.advancedcustomfields.com/resources/register-fields-via-php/

3.30.9.2. Insert Headers and Footers

https://wordpress.org/plugins/insert-headers-and-footers/

3.30.10. Post, Post Type

3.30.10.1. Types - Toolset Types wp:plugin:types
  • WP Types is different! Use types_render_field
  • Database
    • Post types
      wp-types-group
      custom field group
      wp-types-term-group
      custom taxonomy group
      wp-types-user-group
      custom user field group
  • underscores can be - based on how field is named in the plugin

    • Same name in meta_key in wp_postmeta

      • if field is set to not-show on wp-admin, _ is appended
      • Real value is stored as _wpcf-fieldname_in_toolset_ui
      • A multi-value field also has a field _wpcf_fieldname_in_toolset_ui-sort-order
      • The value of field wpcf_fieldname_in_toolset_ui is '' if the field has no value
  • Refer to WP Types Custom Field Checkbox and WP Meta Query
$newsletters = types_render_field('newsletters', array('output'=>'raw'));
// the real field name is actually wpcf-newsletters
// For single line (text) field, use 'output'=>'raw'
// For WYSIWYG field, use 'output'=>'html'. 'raw' might be plain text
// For Image field

$_image = types_render_field('field-image', array('output'=>'raw'));
// Output original full image url
// If 'output' => 'raw', 'size' and 'resize' will be ignored

// use 'url' => 'true' to output the url of the resized image.
$args = array('output'=>'normal', 'url'=>'true', 'width'=> 300, 'resize'=>'proportional');
// Output image url of a fixed width and dynamic height

$args = array('output'=>'normal', 'url'=>'true', 'size'=>'medium', 'resize'=>'proportional' );
// Output image url of a max width and height
// if 'size' is set, 'width' and 'height' will be ignored

/* 
 * 'size' => 'custom_image_size' | 'full' | 'large' | 'medium' | 'thumbnail'
 * Set 'size' and also 'resize' => 'crop' | 'proportional' | 'stretch' | 'pad'
 * If resize is necessary, set 'output' => 'normal'
 * 'padding_color' => 'transparent' | hex when resize=>pad
 */
// For Image field.
  1. Release
    • WP 3.7+
    • 2.0
    • dropped support for PHP 5.2
3.30.10.2. Post Type Switcher wp:plugin:post-type-switcher
  • Switch a post to another post type
3.30.10.3. CPT UI wp:plugin:custom-post-type-ui

Refer to wp:add cpt

  • Remember to refresh permalinks

Premium Custom Post Type UI Extended can show posts of any CPT inside a post using shortcode

Extra CPT support for Genesis and other plugins
https://docs.pluginize.com/article/28-third-party-support-upon-registration
  1. Clear all CPT created by this plugin

    CPT UI > Post Types or Taxonomies > under Import Post Types, add {""} and hit Import

3.30.10.4. Post Expirator wp:plugin:post-expirator
  • Expire posts. Change post status to one of the following:
    • draft
    • trash
    • private
  • Stick or unstick posts
  • Delete posts
  • only support hierarchical taxonomies
  • WP-Cron is used

3.30.11. Backup, migrate, import, export

3.30.11.1. BackUpWordPress wp:plugin:backupwordpress
3.30.11.2. backupbuddy (iThemes) wp:plugin:backupbuddy
3.30.11.3. WP Migrate DB wp:plugin:wp-migrate-db
  • With some string replace function. e.g. //dev.bradt.ca => //bradt.ca, /sites/bradt.ca/public_html => /sites/bradt.ca/public_html
  • wp_options table keep track of recently_edited files that are edited on WP-admin and some other plugins might keep the filepath info
3.30.11.4. wp-migrate-db-pro, wp-migrate-db-pro-media-files

Non-free plugin. Install them in both source and destination websites.

3.30.11.5. wp-all-import-pro, wp-all-export-pro
  1. Basics
    License is for plugin update only
    Both plugins (WP All Import, WP All Export) do not require a license to use
    (no term)
    Create a brand new wp site, use wp cli to import .sql file and install Export-pro
    (no term)
    Export-pro to export parent posts to .csv or .xml
    (no term)
    Install Import-pro to import parent posts
  2. Export-pro wp:plugin:wp-all-export-pro
  3. Import-pro wp:plugin:wp-all-import-pro
    Images
    import them to Media Library and modify the image urls inside body content after posts are imported
    • wp-content/uploads/wpallimport/files
    (no term)
    Taxonomy
    • Taxonomy terms that don't already exist will be created
    Import file
    wp-content/uploads/wpallimport/uploads/xxx/
    (no term)
    Only records with the same unique_key and import id will be updated otherwise they are created
    • Consider creating a custom field for unique id during the first import and use that to identify records for all imports
    • Or overwrite the import file with a new one and run the previous import again
    (no term)
    Import settings > Enable Use StreamReader instead of XMLReader to parse import file
    (no term)
    Db tables
    wp_pmxi_images
    stores the original image urls and the corresponding attachment id
    • id
    • attachment_id
    • e.g. image source url
    • image_filename
    (no term)
    wp_pmxi_posts
    • id
    • bigint(20) unsigned
    • import_id
    • unique_key
    (no term)
    wp_pmxi_imports
    • id
    • d:0
    • e.g. import file name
    (no term)
    wp_pmxi_templates
    • id
    • serialized exported template string
    • d:''
    • import template name
    • Goes on
    (no term)
    For large import, choose an existing import file to creat an import using Manual Schedule, run trigger url and run processing url every 2 mins. Each processing takes about 2 minutes
    (no term)
    Custom Fields
    wpcf fields (toolset)
    wpcf-custom-author where custom-author is the slug in Toolset for the custom field
    WP All Import - ACF Add-on
    wp:plugin:acf

    Cron 

    crontab -e
# */2 * * * * /home/li/c/project/cs-devops/import.sh

    Cron script 

    #!/bin/bash
curl -L "https://li-dev-proj.pantheonsite.io/wp-cron.php?import_key=...&import_id=..&action=processing" -o /home/li/c/project/cs-devops/import.log
    1. Import Settings

      wp-all-import-Import-options.png

    2. Import Options

      wp-all-import-Import-settings.png

3.30.11.6. all-in-one-wp-migration wp:plugin:all-in-one-wp-migration
  • Zero dependency
  • Bypass upload size restriction
  • Support for MySQL and MySQLi
  • wp-cli
  • .wpress files in wp-content/ai1wm-backups with chmod:777
3.30.11.7. wordpress-importer
3.30.11.8. WP Sync DB wp:plugin:wp-sync-db
A fork from wp:plugin:wp-migrate-db
https://github.com/wp-sync-db/wp-sync-db
Installation
requires wp:plugin:github-updater
  • Just download the zip
  • copy the connection info from the source website and paste into the destination website in order to pull to destination

3.30.12. DB Optimize

3.30.12.1. wp-optimize

3.30.13. WP Admin

3.30.13.1. duplicate-post wp:plugin:duplicate-post
  • Clone post
3.30.13.2. Admin Columns wp:plugin:codepress-admin-columns
3.30.13.3. tinymce-advanced
  • Add, remove and arrange buttons shown on the Visual Editor toolbar.
  • Enable or disable 15 plugins for TinyMCE.
  • Modify options such as keeping the paragrah tags in the Text editor and importing CSS classes from the theme's editor-style.css
  • Always update as wp core updates
3.30.13.4. disable-embeds wp:plugin:disable-embeds
Disable embed, oEmbed and remove wp-embed.min.js
https://kinsta.com/knowledgebase/disable-embeds-wordpress/
(no term)
Refer to wp:api:oembed

3.30.14. Cache, CDN

3.30.14.1. wp-redis wp:plugin:wp-redis
3.30.14.2. Super Cache (SuperCache)

Warning: include(/var/www/html/wp-content/plugins/wp-super-cache/wp-cache-base.php): failed to open stream

If you encounter this problem, your WP_CACHE cache folder is not set properly. Check `wp-config.php`. You should see 

define('WP_CACHE',true);
define('WPCACHEHOME', ABSPATH. 'wp-content/plugins/wp-super-cache/');

There's another place where the cache folder is defined: `$cache_path` in `wp-content/wp-cache-config.php`

Uninstall To manually uninstall:

• Turn off caching on the plugin settings page and clear the cache. • Deactivate the plugin on the plugins page. • Remove the WP_CACHE define from wp-config.php. It looks like define( 'WP_CACHE', true ); • Remove the Super Cache mod_rewrite rules from your .htaccess file. • Remove the files wp-content/advanced-cache.php and wp-content/wp-cache-config.php • Remove the directory wp-content/cache/ • Remove the directory wp-super-cache from your plugins directory.

If all else fails and your site is broken • Remove the WP_CACHE define from wp-config.php. It looks like define( 'WP_CACHE', true ); • Remove the rules (see above) that the plugin wrote to the .htaccess file in your root directory. • Delete the wp-super-cache folder in the plugins folder. • Optionally delete advanced-cache.php, wp-cache-config.php and the cache folder in wp-content/.

3.30.14.3. W3 Total Cache wp:plugin:w3-total-cache
  1. Manually remove

    Files :: global search W3TC_ADDIN_FILE_

    Backup and remove wp-content/plugins/w3-total-cache Rename or remove wp-content/cache Rename or remove wp-content/w3tc-config and maybe wp-content/w3tc Rename or remove Files above under wp-content root :: advanced-cache.php db.php object-cache.php and maybe wp-total-cache-config.php Remove lines related to w3tc in .htaccess Remove line from wp-config.php define('WP_CACHE', true)

  2. Releases

    0.9.5.1 > 0.9.6 > 0.9.7

3.30.14.4. WP Rocket wp:plugin:wp-rocket
  • https://wp-rocket.me by WP Media
  • Plans: features are the same for all plans
    • Single: 1 site
    • Plus: 3 sites
    • Infinite: unlimited sites
3.30.14.5. cloudflare

https://www.digitalocean.com/community/tutorials/how-to-mitigate-ddos-attacks-against-your-website-with-cloudflare

It includes CDN, SSL, static cotnent caching, firewall and traffic analytics tools.

On CloudFlare

  • Add a website
  • confirm DNS records are correct
  • Change DNS namerservers to cloudflare NS
  • CloudFlare is acting as a reverse proxy to your website

Initial setup https://support.cloudflare.com/hc/en-us/articles/201897700

  • On web server, whitelist CloudFlare IP addresses
  • Need to install mod_cloudflare Apache module in order for web server to see visitor IP
  • Firewall > IP Firewall > Access Rules
  1. Turn off Cloudflare for a domain

    Sometimes, you want to bypass Cloudflare for a domain e.g. abc.ca and www.abc.ca.

    • e.g. for the first time to use Let's Encrypt (certbot) with Nginx to setup redirect https://abc.ca to https://xyz.ca and both of those websites are not hosted on Nginx
    • DNS > Turn off for those domains
    • Crypto > SSL set to Off
  2. letsencrypt:certbot

    CloudFlarec.com > Crypto

    • SSL set to Full or Full (strict)
    • Disable Always Use HTTPS
  3. TS: Restart Nginx Cause Error 525: SSL Handshake Failed

    The SSL certificate on Cloudflare needs to renew

    • Cloudflare dashboard for the website > Crypto > at the very bottom Disable Universal SSL
    • Wait 5 minutes and turn it back on
    • Still on Crypto page, under SSL > Universal SSL Status, wait until it says Active Certificate (usually it takes about 10 minutes)

    Or maybe the target Nginx server's SSL certificate has expired e.g. Nginx is used to redirect this website to another website. Both websites are not hosted on Nginx

  4. TS: ERR_TOO_MANY_REDIRECTS

    When Crypto > SSL is set to Flexible, Cloudflare sends all requests between Cloudflare and the origin web server over HTTP non-encrypted. Your Nginx receives HTTP request and might have redirect rules to redirect to https and thus causes redirect loop.

  5. TS: err_ssl_version_or_cipher_mismatch

    This is due to the SSL given by Cloudflare is not active yet. Wait for 10 minutes.

3.30.15. Google

3.30.15.1. Google Analytics Dashboard for WP by ExactMetrics (GADWP) wp:plugin:google-analytics-dashboard-for-wp
  • Show stats for each post!
  • Insert GA tracking code dynamically using analytics.js
  • Can add events for downloads, mailto, telephone and outbound links
  • Can exclude by user roles for events
  1. Hook for Commands
    $commands = [
    // command 1
    [
        'command'      => 'create',
        'fields'       => [
            'trackingId'   => 'UA-xxx',
            'cookieDomain' => 'auto',
        ],
        'fieldsobject' => [],
    ],
    // command 2
    [
        'command'      => 'send',
        'fields'       => [
            'hitType' => 'pageview',
        ],
        'fieldsobject' => null,
    ]
];

// do_action( 'gadwp_analytics_commands', $this);
add_action( 'gadwp_analytics_commands', function( $gadwp ) {
    $commands = $gadwp->get();

    // var_dump($commands);

    $last = array_pop( $commands ); // last command always is send pageview

    global $post;
        $fields = [];
        $fields['option'] = 'contentGroup1';
        $fields['value'] = esc_attr($post->post_type);
        $fieldsobject = null;
        $commands[]   = $gadwp->prepare( 'set', $fields, $fieldsobject ); // Add a new line to the commands array
    // ga('set', 'contentGroup1', '<Group Name>');
        $commands[]   = $last;

        $gadwp->set($commands); // Store the new commands array
});

3.30.16. SEO

3.30.16.1. Yoast SEO wordpress-seo wp:plugin:wordpress-seo
How to SEO?
https://yoast.com/wordpress-seo/
API and fitlers
https://yoast.com/wordpress/plugins/seo/api/
Link Google Search Console
google:search console
(no term)
Yoast SEO Premium: paid by number of sites
  • https://yoast.com/wordpress/plugins/seo/
  • Keywords, keyphrases, synonyms, related keywords and all word forms for those
    Free
    only 1 keyword or keyphrase
  • Warning on most important pages, and when these are not updated for 6 months
  • Get suggestions for links to other pages
  • Show 5 words or phrases used the most on a page
  • Redirect manager
  • Focus keyword export
  • Google, Facebook and Twitter previews
    Free
    only Google preview
  • Both versions
    • schema.org
    • Flesch Reading Ease score
    • Primary category
    • Canonical URL
  1. Version

    7 - UI

  2. Open Graph wp:plugin:wordpress-seo:og
    • At least 200x200 pixels image for og:image
    • Resave the post/article on WP first to prevent WP Super Cache as FB crawler gets to the page differently and then run Facebook Open Graph Object Debugger
    • Refer to html:og
    • required
    • e.g. article, for archive page, it's object
    • required
    • required
    • { "locale": "en_us" }
    • og:description
    • og:site_name
    • og:updated_time
    <!-- Archive Page -->
<meta property="og:locale" content="en_US" />
<meta property="og:type" content="object" />
<meta property="og:title" content="Archive Page Title" />
<meta property="og:url" content="https://www.a.ca/category/a-category/" />
<meta property="og:site_name" content="My Site Name" />
<meta property="og:image" content="https://www.a.ca/wp-content/uploads/2016/12/site-image-og-image.png" />
<meta property="og:image:secure_url" content="https://www.a.ca/wp-content/uploads/2016/12/site-image-og-image.png" />
<meta property="og:image:width" content="470" />
<meta property="og:image:height" content="351" />

<!-- Single Post Page -->
<meta property="og:locale" content="en_US" />
<meta property="og:type" content="article" /> <!-- article is a namespace, all single posts use article -->
<meta property="og:title" content="Post Title" />
<meta property="og:description" content="Post Excerpt By Default" />
<meta property="og:url" content="https://www.a.ca/features/a-feature-post/" />
<meta property="og:site_name" content="My Site name" />
<meta property="article:publisher" content="https://www.facebook.com/my-fb-name" />
<meta property="article:tag" content="A Tag name" /> <!-- multiple tags -->
<meta property="article:section" content="Primary Category Name" /> <!-- Primary category -->
<meta property="article:published_time" content="2019-04-05T19:39:44+00:00" />
<meta property="article:modified_time" content="2019-04-08T19:16:37+00:00" />
<meta property="og:image" content="https://www.a.ca/wp-content/uploads/2016/12/site-image-og-image.png" /> <!-- Post Featured Image, if not, use site image -->
<meta property="og:image:secure_url" content="https://www.a.ca/wp-content/uploads/2016/12/site-image-og-image.png" />
<meta property="og:image:width" content="470" />
<meta property="og:image:height" content="351" />
  3. Turn off json-ld wp:plugin:wordpress-seo:json-ld

    Once SEO > Search Appearance > General > Knowledge Graph > Company name and logo are specified, Plugin inserts Oranization and Website by default 

    {
  "@context": "http:\/\/schema.org",
  "@type": "WebSite",
  "@id": "#website",
  "url": "https:\/\/mywebsite.com\/",
  "name": "My Website Name",
  "potentialAction": {
    "@type": "SearchAction",
    "target": "https:\/\/mywebsite.com\/?s={search_term_string}",
    "query-input": "required name=search_term_string"
  }
}

    {
  "@context": "http:\/\/schema.org",
  "@type": "Organization",
  "url": "https:\/\/mywebsite.com\/",
  "sameAs": [],
  "@id": "#organization",
  "name": "My Organization Name",
  "logo": "http:\/\/mywebsite.com\/wp-content\/uploads\/2018\/02\/logo.jpg"
}

    Use filter to turn it off and customize it https://www.bybe.net/yoast-seo-guide-disable-schema-json-ld-wordpress/ 

    function bybe_remove_yoast_json($data){
  $data = array();
  return $data;
}
add_filter('wpseo_json_ld_output', 'bybe_remove_yoast_json', 10, 1);
  4. Turn off canonical

    add_filter( 'wpseo_canonical', '__return_false' );

  5. Setting: add exclude post from sitemap
    Edit a post
    Setting > Allow search engines to show this Post in search results
    (no term)

    Theme functions.php 

    add_filter( 'wpseo_exclude_from_sitemap_by_post_ids', function ($post_ids) {
    if (function_exists('wc_get_page_id')) {
        $post_ids[] = wc_get_page_id( 'checkout' );
    }
    return $post_ids;
} );
  6. Setting: turn off Author Archives

    SEO > Search Appearance > Archives > disable Author archive Otherwise /author-sitemap.xml will show the admin name

  7. Release

    1.5.2.7, 7.1, 7.5

3.30.16.2. all-in-one-seo-pack wp:plugin:all-in-one-seo-pack
  • By ServMask
  1. Release

    2.3.11.2 2.4.5 :: WP v4.0 to 4.4 2.4.6.1

3.30.17. Social media, Push Notification

3.30.17.1. WordPress to Buffer Pro
$199 for lifetime unlimited websites or $89/yr for unlimited websites
https://www.wpzinc.com/plugins/wordpress-to-buffer-pro/
(no term)
Documentaion
(no term)
Compatible with wp:plugin:acf, wp:plugin:the-events-calendar
(no term)
WP-CLI to repost
Chrome Extension
share any url, scan page to get images, specify which social accounts to post and add images
3.30.17.2. Simple Social Icons wp:plugin:simple-social-icons
3.30.17.3. OneSignal wp:plugin:onesignal-free-web-push-notifications
  • Free to web push notifications for up to 30k subscribers and no limit on number of push notifications you can send
  • Custom Code can be selected on OneSignal.com. It doesn't have to be WordPress Plugin or Website Builder

3.30.18. Post type - Comments

3.30.18.1. Disqus

If you receive a lot of errors about dsq_sync_forum in Apache logs, it is caused by Disqus tries to sync with WP comments so frequently.

First in WP Disqus setting, check "Disable automated comment importing"

Empty all cron rows in wp-options table `UPDATE wp_options SET option_value='' WHERE option_name='cron';"`

https://wordpress.org/support/topic/high-server-load-and-dsq_sync_forum-problem?replies=6

3.30.18.2. disable-comments wp:plugin:disable-comments
Keywords
disable comments, remove comments
(no term)
Turn off comments and it requires comment to be deleted

If you want to keep comments in db, use this custom code without using the plugin. 

function filter_media_comment_status( $open, $post_id ) {
  $post = get_post( $post_id );

  if( in_array($post->post_type, array('attachment', 'post')) ) {
   return false;
  }
  // or simply return false;
  return $open;
}
add_filter( 'comments_open', 'filter_media_comment_status', 10 , 2 );

// other actions might be needed

Because in themes 

if( comments_open() || get_comments_number() ) {
  // open or there's at least one comment
  comments_template();
}

// or for custom post types
if( post_type_supports( get_post_type(), 'comments' ) ) {
    // put comment related code here, including...
    if( comments_open() ) {
        // ...a comment form, maybe
    }
}
else {
    // sit back and relax
}
  • WP Admin > Settings > Discussion > disable Allow People to post comments on new articles for future posts and disable Allow link notifications from other blogs (pingbacks and trackbacks) on new articles
  • Maybe useful to set Automatically close comments on articles older than 0 days

3.30.19. Post type - Events

3.30.19.1. The Events Calendar wp:plugin:the-events-calendar
  • Free
    • Month View
    • List View
    • Day View
    • Saved Content: saved Venues and Organizers
    • Keyword Search
    • AJAX
    • iCal & Gcal Export
    • Events List Widget
  • Pro: https://theeventscalendar.com/product/wordpress-events-calendar-pro/ by Modern Tribe
    • Recurring events
    • Week View
    • Photo View
    • Map View
    • Location Search
    • Venue & Organizer View
    • Advanced Widgets
      • Mini Calendar Grid
      • Advanced Upcoming Events list
      • Featured Venue
      • Countdown
    • Shortcodes
    • Additional Fields
3.30.19.2. all-in-one-event-calendar

3.30.20. Post type - Testimonial

3.30.20.1. Strong Testimonials wp:plugin:strong-testimonials

3.30.21. Forms, Polls, Surveys

3.30.21.1. Gravity Forms wp:plugin:gravity forms
  1. Form setting
    Allow field to be populated dynamically
    so that populating this form field with a value using query string or hook.
    (no term)
    Deleting a field will also delete all entry data associated with that form field
    (no term)
    Input Mask
    • Use a ‘9’ to indicate a numerical character
    • Use a lower case ‘a’ to indicate an alphabetical character
    • Use an asterick ‘*’ to indicate any alphanumeric character
    • Use a question mark ‘?’ to indicate optional characters. Note: All characters after the question mark will be optional
    • All other characters are literal values and will be displayed automatically
    • Examples
      Date
      99/99/9999
      Course code
      aaa 999
      License Key
      ***-***-***
    (no term)
    Post fields enable you to create WP Post
  2. Shortcodes & Merge Tags

    https://docs.gravityforms.com/shortcodes/

    1. [gravityform id="1"] wp:plugin:gravity forms:sc:gravityform
      [gravityform id="1" title="false" description="false"]
[gravityform id=1 field_values='parameter_name1=value1&parameter_name2=value2']
      id
      required
      title
      true/false defaut:true
      description
      true/false default:true
      ajax
      true/false
      tabindex
      int
      field_values
      populate values
    2. [gravityforms action="conditional"]

      Insert message content to Admin and User Notification emails as well as the Confirmation Message that is displayed when a form is submitted 

      <!-- Only one condition is allowed -->
[gravityforms action="conditional" merge_tag="{Number:1}" condition="greater_than" value="10"]
This content would be displayed if the value of field id 1 is greater than 10.
[/gravityforms]

[gravityforms action="conditional" merge_tag="{Country:5}" condition="is" value="United States"]
This content would be displayed if the value of field id 5 is United States.
[/gravityforms]

[gravityforms action="conditional" merge_tag="{Country:5}" condition="isnot" value=""]
This content would be displayed if no value exists for field id 5.
[/gravityforms]
    3. Actions [gravityform action="*"]

      Display details from user meta fields 

      <!-- Display details from user meta fields. Requires User Registration Add-On  -->
[gravityform action="user" key="ID" output="raw" /]


<!-- Displays the login form. Requires User Registration Add-On -->
[gravityform action="login" description="false" logged_in_message="Yay! You are logged in!" registration_link_text="Register for my super awesome site" forgot_password_text="Stop forgetting your password" /]
    4. Merge Tags

      Populate submitted field values in notification emails, post content templates and more. https://docs.gravityforms.com/merge-tags/

  3. Functions
    1. gravity_form wp:plugin:gravity_forms:f:gravity_form
    2. gravity_form_enqueue_scripts
      Placement
      before wp_head() is called
      gravity_form_enqueue_scripts( $form_id, $is_ajax = false);
  4. Hooks and actions

    https://docs.gravityforms.com/category/extending-gravity-forms/hooks/actions/

    1. gform_field_value_$parameter_name wp:plugin:gravity forms:filter:gform_field_value_$parameter_name

      Populate any field with parameter your_parameter with the result of the function my_custom_population_function 

      add_filter( 'gform_field_value_your_parameter', 'my_custom_population_function' );
function my_custom_population_function( $value, $field, $name ) {
    return 'boom!';
    // populate from a cookie
    return $_COOKIE['utm_campaign'];
    // populate a list field
    // new way
    $list_array = array(
        array(
            'Column 1' => 'row1col1',
            'Column 2' => 'row1col2',
            'Column 3' => 'row1col3',
        ),
        array(
            'Column 1' => 'row2col1',
            'Column 2' => 'row2col2',
            'Column 3' => 'row2col3'
        ),
  );
    return $list_array;

    // old way
    return array( 'row 1 - col1', 'row 1 - col2', 'row 1 - col3',
        'row 2 - col1', 'row 2 - col2', 'row 2 - col3',
        'row 3 - col1', 'row 3 - col2', 'row 3 - col3' );
}

// Populate multiple fields using one function
add_filter( 'gform_field_value', 'populate_fields', 10, 3 );
function populate_fields( $value, $field, $name ) {

    $values = array(
    'field_one'   => 'value one',
    'field_two'   => 'value two',
    'field_three' => 'value three',
    );

    return isset( $values[ $name ] ) ? $values[ $name ] : $value;
}

// Populate a time field
add_filter( 'gform_field_value_time', 'populate_time' );
function populate_time( $value ) {
    $local_timestamp = GFCommon::get_local_timestamp( time() );

    return date_i18n( 'h:i A', $local_timestamp, true );

        // Populate a fixed date
        return '10/10/2010';
}

// Populate author email
add_filter( 'gform_field_value_author_email', 'populate_post_author_email' );
function populate_post_author_email( $value ) {
    global $post;

    $author_email = get_the_author_meta( 'email', $post->post_author );

    return $author_email;
}
    2. gform_field_container gravityforms:gform_field_container
      add_filter( 'gform_field_container', 'add_bootstrap_container_class', 10, 6 );
function add_bootstrap_container_class( $field_container, $field, $form, $css_class, $style, $field_content ) {
    $id = $field->id;
    $field_id = is_admin() || empty( $form ) ? "field_{$id}" : 'field_' . $form['id'] . "_$id";
    return '<li id="' . $field_id . '" class="' . $css_class . ' form-group">{FIELD_CONTENT}</li>';
}
  5. Query String

    Populate form fields http://siteurl.com/form-url/?your_parameter=value

  6. Events

    https://docs.gravityforms.com/gravity-forms-notification-events/

  7. CSS Styling

    https://docs.gravityforms.com/basic-structure/ https://docs.gravityforms.com/css-targeting-examples/ https://docs.gravityforms.com/css-visual-guide/ https://docs.gravityforms.com/css-ready-classes/ 

    <div class="gform_body">
    <ul id="gform_fileds_1" class="gform-fields top_label form_sublabel_below field_description_below">
    <!-- field #1 -->
    <li id="field_1_7" class="gfield field_sublabel_below field_description_below gfield_visibility_visible">
        <label for="input_1_7" class="gfield_label">Label 1</label>
        <div class="ginput_container ginput_container_text">
        <input type="text" name="input_7" id="input_1_7" value class="medium" tabindex="1" aria-invalid="false">
        </div>
    </li>

    <!-- field #2 -->
    </ul>
</div>
  8. Add-Ons
    1. User Registration

      A feed is used to pass form submission info over to the add-on. In Form Setting > User Registration > Add New Feed

    2. Polls - gravityformspolls
  9. WP Config

    https://docs.gravityforms.com/wp-config-options/ 

    define( 'GF_LICENSE_KEY', 'YOUR-LICENSE-KEY-HERE' );
  10. Database
    wp_gf_form
    forms
  11. TS: jQuery is not found

    Gravity Forms by default will write an inline JavaScript call to jQuery on every form you add to a page. This will throw an error if you’re loading jQuery in the footer of your site (which you should be doing). 

    add_filter("gform_init_scripts_footer", "init_scripts");
function init_scripts() {
    return true;
}
3.30.21.2. Gravity Perks
3.30.21.3. Crowdsignal wp:plugin:crowdsignal
  • https://crowdsignal.com by Automattic. Formerly Polldaddy

    • Plan

      • Pro: $17/month or $200/yr
      • Unlimited polls and surveys
      • 1 user account
      • No Crowdsignal branding
      • Can't customize domain
      • 10k email/month invites
    • Sharing
      • Permalink
      • iframe, HTML/JavaScript, WordPress shortcode (WP Crowdsignal plugin), QR Code
      • Send emails (in Email Groups) to invite them to take a survey
    • Receive GDPR erasure requests from receivers who want to delete their responses and votes
    • Email Group
      • Every member/contact in any email group has 4 fields: email address, first name, last name and custom data
  • ask one simple question
3.30.21.4. contact-form-7
  1. Shortcode

    Add html id 

    contact-form-7 id="1234" title="Contact form 1" html_id="contact-form-1234" html_class="form contact-form"]
  2. autop wp:p:contact-form-7:autop

    wp-config.php 

    define( 'WPCF7_AUTOP', false );
  3. hooks - reCaptcha

    Default template 

    [response]

<p><label for="your-name">Your Name (required)</label><br>
[text* your-name id:your-name]</p>

<p><label for="your-email">Your Email (required)</label><br>
[email* your-email id:your-email]</p>

<p><label for="your-message">Your Message</label><br>
[textarea* your-message id:your-message]</p>

<p>[submit "Send"]</p>

    Turn off wp:p:contact-form-7:autop

    1. filter:wpcf7_form_elements

      Remove <noscript> for reCaptcha which has an iframe that is not WCAG ready 

      function csWpcf7FormElements($form) {
  try {
    $dom = new DOMDocument;
    $dom->loadHTML($form);
    $noscripts = $dom->getElementsByTagName('noscript');
    while ($noscripts->length > 0) {
  $i = $noscripts->item(0);
  $i->parentNode->removeChild($i);
    }
    return $dom->saveHTML();
  }
  catch (Exception $err) {
    return $form;
  }

  return $form;

}
add_filter( 'wpcf7_form_elements', 'csWpcf7FormElements');
    2. filter:shortcode_atts_wpcf7 wp:p:contact-form-7:f:shortcode_atts_wpcf7
      add_filter( 'shortcode_atts_wpcf7', 'custom_shortcode_atts_wpcf7_filter', 10, 3 );

function custom_shortcode_atts_wpcf7_filter( $out, $pairs, $atts ) {
    $my_attr = 'destination-email';

    if ( isset( $atts[$my_attr] ) ) {
    $out[$my_attr] = $atts[$my_attr];
    }

    return $out;
}
    3. action:wpcf7_before_send_mail wp:p:contact-form-7:a:wpcf7_before_send_mail
  4. DOM Events

    https://contactform7.com/dom-events/

    wpcf7invalid — Fires when an Ajax form submission has completed successfully, but mail hasn’t been sent because there are fields with invalid input. wpcf7spam — Fires when an Ajax form submission has completed successfully, but mail hasn’t been sent because a possible spam activity has been detected. wpcf7mailsent — Fires when an Ajax form submission has completed successfully, and mail has been sent. wpcf7mailfailed — Fires when an Ajax form submission has completed successfully, but it has failed in sending mail. wpcf7submit — Fires when an Ajax form submission has completed successfully, regardless of other incidents.

    event.detail detail.inputs detail.contactFormId The ID of the contact form. detail.pluginVersion The version of Contact Form 7 plugin. detail.contactFormLocale The locale code of the contact form. detail.unitTag The unit-tag of the contact form. detail.containerPostId The ID of the post that the contact form is placed in.

    Example redirect to a page 

    document.addEventListener( 'wpcf7mailsent', function( event ) {
  var inputs = event.detail.inputs;
  if ( '123' == event.detail.contactFormId) {
    location = 'http://example.com/';
  }
}, false );

document.addEventListener( 'wpcf7submit', function( event ) {
    var inputs = event.detail.inputs;

    for ( var i = 0; i < inputs.length; i++ ) {
    if ( 'your-name' == inputs[i].name ) {
        alert( inputs[i].value );
        break;
    }
    }
}, false );
  5. Form tags, akismet
    Select Country (required)
[checkbox* your-country use_label_element "China" "India" "San Marino"]

Select Sports
[radio your-sports label_first default:2 "Football" "Tennis" "Pole-vault"]

Select Fruit (required)
[checkbox* your-fruit exclusive "Apple" "Banana" "Grape"]

Select Browser (required)
[select* your-browser include_blank "Firefox" "Safari" "Opera" "Lynx"]

Select Ghkdsjdf
[select your-ghkdsjdf multiple "fsdfs" "klgjfk" "vdrie"]

[submit "Send"]
    1. Akismet

      akismet:author Add this option to the field that accepts the name of the sender. Example: [text* your-name akismet:author] akismet:author_email Add this option to the field that accepts the email address of the sender. Example: [email* your-email akismet:author_email] akismet:author_url Add this option to the field that accepts the URL of the sender. Example: [text your-url akismet:author_url]

      Test viagra-test-123 as name

    2. File upload

      https://contactform7.com/file-uploading-and-attachment/ 

      [file your-file filetypes:pdf|txt limit:2mb]
//id:foo
//limit:1048576
//limit:1024kb
//limit:1mb

// multiple classes
// class:y2008 class:m01 class:d01

      Default file upload location is wp-content/uploads/wpcf7_uploads

    3. select with pipes
      [select your-recipient "CEO|ceo@example.com"
                    "Sales|sales@example.com"
                    "Support|support@example.com"]

      Only the part before pipes are revealed on the front end to prevent spam. Use [your-recipient] in the Mail template. To get value before pipe, [_rawfield name] e.g. [_raw_your-recipient]

    4. Attribute - default
      // The field will obtain its default value from the GET variable with the same name (“your-name”)
// http://example.com/contact/?your-name=John+Smith
// John Smith
[text* your-name default:get]

// This form-tag has two default options and a value “Your Name”. The options are evaluated from the first to the last. In this example, default:get is evaluated first. If the “your-name” GET variable has a value, it will be used for the default value. If that value is empty, default:post_meta will be evaluated next. If both of these options do not have values, “Your Name” will be used.
[text* your-name default:get default:post_meta "Your Name"]

      Logged-in user fields 

      Your Name: [text* your-name default:user_display_name]
Your E-mail: [email* your-email default:user_email]

      Get default from shortcode attributes 

      [contact-form-7 id="123" title="Contact Form" destination-email="xxxxxx@example.com"]

[email* destination-email default:shortcode_attr]

// Filter needs to setup to catch custom attributes. Refer to wp:p:contact-form-7:f:shortcode_atts_wpcf7
    5. hidden fields

      id:foo, 

      [hidden your-text class:y2008 class:m01 class:d01]
[hidden your-email default:user_email "example@example.com"]
  6. multifile-upload-field-for-contact-form-7

    Form tag [multifile your-photos] to upload multiple files as one zip file

  7. contact-form-7-confirm-email-feild

    Confirm email field 

    [email* your-email akismet:author_email]
<label> <p>Confirm Email:</p> [confirm_email* confirm-email]</label>
  8. flamingo

    save form submission By default, flamingo will take [your-subject] [your-name] [your-email]. Overwrite them

    Additional Settings in the contact form 

    flamingo_email: "[the-email-field]"
flamingo_name: "[the-name-field]"
flamingo_subject: "[the-subject-field]"
  9. contact-form-plugin

    receive messages from customers to your email addresses. Download, activate and paste [bestwebsoft_contact_form] shortcode on any page, post or widget to display the form. Customize the form styles and contents easily with the pre-build options.

  10. Release

    3.7.2 4.0 :: WP 3.9+ 4.1 :: WP 4.0+ 4.9 :: WP 4.7+ 5.0 :: WP 4.8+, on_sent_ok and on_submit are removed 5.0.1

3.30.21.5. constant-contact-forms

Require WP 4.0+ and PHP 5.4+ Widget: Constant Contact Form shortcode: [ctct form="5872"]

  1. Setting

    Enable logging

  2. wp filters

    Add label on ConstantContact.com for custom field values add_filter( 'constant_contact_include_custom_field_label', '__return_true' );

  3. Caveats

    There's no name field only first name and last name field captured on ConstantContact.com

    If there're more than one Constant Contact forms, the form fields will have the same id. This will create a problem when clicking a later opt-in checkbox label always jump to the first opt-in checkbox

    Make opt-in checkbox required and fix opt-in checkbox label 

    $(function(){
    $('label[for=ctct-opt-in]').prev().addClass('ctct-form-field-required').prop('required',true);;

    $('label[for=ctct-opt-in]').on('click', function(e){
  e.preventDefault();
  e.stopPropagation();
  var $input = $( this ).prev(); // previous sibling
  $input.prop('checked', !$input.prop("checked")); // jQuery 1.6+
    });

});
3.30.21.6. constant-contact-forms-by-mailmunch

It takes way too much work to customize the form..

3.30.22. Site search

3.30.22.1. WP Advanced Search wp:plugin:wpas

https://github.com/raideus/wp-advanced-search http://wpadvancedsearch.com/docs/setup/

  • Copy code to wp-content/mu-plugins/wp-advanced-search
  • wp-content/mu-plugins/load-wpas.php
define( 'WPAS_URI', WPMU_PLUGIN_URL . '/wp-advanced-search' );
require_once( 'wp-advanced-search/wpas.php' );

// define your forms
function my_search_form() {
    $args = array();
    $args['wp_query'] = array('post_type' => 'post',
              'posts_per_page' => 5);
    $args['fields'][] = array('type' => 'search',
              'title' => 'Search',
              'placeholder' => 'Enter search terms...');
    $args['fields'][] = array('type' => 'taxonomy',
              'taxonomy' => 'category',
              'format' => 'select');
    register_wpas_form('my-form', $args);    
}
add_action('init', 'my_search_form');
  1. Fields

    http://wpadvancedsearch.com/docs/template-setup/appearance/ 

    // pre_html and post_html
$args['fields'][] = array(
    'type' => 'search',
    'placeholder' => 'Enter search terms...',
    'pre_html'    => '<div class="row">',
    'post_html'   => '</div>'
);

// class
$args['fields'][] = array(
    'type'        => 'search',
    'placeholder' => 'Enter search terms...',
    'class'       => array( 'myclass', 'anotherclass' )
);

// use an html-type field to display messages
$args['fields'][] = array(
    'type'  => 'html',
    'value' => '<div class="my-element"><h3>Some custom HTML content here</h3></div>'
);

// disable wrapper for the whole form
$args['form'] = array( 'disable_wrappers' => true );

// works well with js:lib:select2
$args['fields'][] = array(
    'type'  => 'html',
    'value' => '<div class="form-row">',
);

$args['fields'][] = array(
    'type'        => 'meta_key',
    'meta_key'    => 'wpcf-listing-city',
    'placeholder' => 'City...',
    'format'      => 'text',
    'class'       => array( 'col-auto', 'form-control form-control-lg' ),
    'pre_html'    => '<div class="form-group d-none d-md-block col-md-4">',
    'post_html'   => '</div>',
    'allow_null'  => 'ALL',
    'operator'    => 'AND',
    'data_type'   => 'CHAR',
    'compare'     => 'LIKE',
);

// meta_key with select dropdown
$query = "
    SELECT DISTINCT CAST(pm.meta_value AS DATE)
    FROM wp_posts p
   INNER JOIN wp_postmeta pm ON pm.post_id = p.id
    WHERE p.post_type = 'agenda'
  AND p.post_status = 'publish'
  AND pm.meta_key = 'agenda_date'
  ORDER BY CAST(pm.meta_value AS DATE)
";
global $wpdb;
$agenda_date = $wpdb->get_col($query);

$agenda_date_form = array();
foreach($agenda_date as $d) {
    $agenda_date_form[$d] =$d;
}

$args['fields'][] = array(
    'type'      => 'meta_key',
    'meta_key'  => 'agenda_date',
    'format'    => 'select',
    'data_type' => 'DATE',
    'values'    => $agenda_date_form,
    'allow_null' => 'Select a date', // treat the first key/value pair in values as null
    'class'     => array( 'col-auto', 'form-control form-control-lg' ),
    'pre_html'  => '<div class="form-group col-md-6">',
    'post_html' => '</div>',
);

$args['fields'][] = array(
    'type'       => 'taxonomy',
    'taxonomy'   => 'listing_category',
    'format'     => 'multi-select',
    'class'      => array( 'col-auto', 'select2 form-control form-control-lg' ),
    'pre_html'   => '<div class="form-group d-none d-md-block col-md-9">',
    'post_html'  => '</div>',
    'allow_null' => 'ALL',
    'nasted'     => true,
    'operator'   => 'IN'
);
$args['fields'][] = array(
    'type'  => 'html',
    'value' => '</div>',
);
  2. $args['wp_query']
  3. Other configuration

    http://wpadvancedsearch.com/docs/config/

  4. Form configuration $args['form']

    http://wpadvancedsearch.com/docs/config/form-config/ 

    $args['form'] = array( 
  // arguments here
);

    AJAX form http://wpadvancedsearch.com/docs/config/form-config/ajax/ 

    $args['form']['ajax'] = array(
// arguments here
);
  5. Display forms
    $my_search = new WP_Advanced_Search('my-form');

// display form

// ob_start();
$my_search->the_form();
// $form_output = ob_get_clean();
// $form_output = sprintf( '<div class="row"><div class="col p-4">%s</div></div>', $form_output);
// echo $form_output

// display search result
$query = $my_search->query();

// you can also modify the global $wp_query
// global $wp_query;
// $temp = $wp_query;
// $wp_query = $search->query();
// run main loop e.g. if (have_posts()) { ... }
// wp_reset_query();
// $wp_query = $temp; // this might not be necessary

if ( $query->have_posts() ): 
    while ( $query->have_posts() ): $query->the_post();
the_title();
endwhile;
endif;

// pagination :: use the same arguments wp core pageinate_links()
// https://codex.wordpress.org/Function_Reference/paginate_links
// Refer to wp:theme:pagination 

$my_search->pagination(array('prev_text' => '«','next_text' => '»'));

// result count

$args = array(); 
echo 'Displaying results ' . $my_search->results_range($args) . ' of ' . $wp_query->found_posts;

//pre – Content to display before the output
//marker – Content to use as the range marker (ie passing a hyphen “-” would show the range as “1-5″)
//post – Content to display after the output
$args = array('pre' => '',
                            'marker' => '-',
                            'post' => '');
  6. Alter query with pre_get_post
    add_action( 'pre_get_posts', 'lili_pre_get_posts' );
function lili_pre_get_posts( $query ) {
    if (! is_admin()) {
        if ( $query->is_main_query() ) {
            // WPAS is not a main query
        }
        else {
            // not main query
            // Agenda Archive display with WPAS
            if ( in_array( $query->get( 'post_type' ), array( 'agenda' ) ) ) {
                // $query->set( 'posts_per_page', '-1' ); // this was set in WPAS $args['wp_query']['posts_per_page'] = -1

                // need to add clauses in order to sort custom fields
                $meta_query = array(
                    'date_clause'  => array(
                        'key'     => 'agenda_date',
                        'compare' => 'EXISTS',
                    ),
                    'start_clause' => array(
                        'key'     => 'agenda_start_time',
                        'compare' => 'EXISTS',
                    ),
                );

                // merge WPAS meta query
                $old_meta_query = $query->get('meta_query');
                $old_meta_query = (empty($old_meta_query)) ? array() : $old_meta_query;
                $meta_query = array_merge($meta_query, $old_meta_query);

                $query->set( 'meta_query', $meta_query );
                $query->set( 'orderby', array(
                    'date_clause'  => 'ASC',
                    'start_clause' => 'ASC',
                    'date'         => 'ASC',
                ) );
            }
        }
    }
    return $query;
}
3.30.22.2. custom-search-plugin

Add custom post types and taxonomies to WordPress website search results. A quick and easy way to search everything within custom post types and taxonomies.

3.30.23. Redirect, Permalinks

3.30.23.1. custom-permalinks

Change URL to any structure for individual post, page, tag or category. Need to resave Permalinks setting

3.30.23.2. redirection wp:plugin:redirection

3.30.24. Security

3.30.24.1. hc-custom-wp-admin-url
3.30.24.2. better-wp-security (iThemes Security)

https://en-ca.wordpress.org/plugins/better-wp-security/

Latest plugin version usually requires the latest WP Core. PHP version 5.2+ Site has to run either on Apache, LightSpeed or Nginx Work on multi-site

Will add lines to .htaccess and wp-config.php

Banned Users :: get blacklisted IPs, user agents and apply to your website. Enable HackRepair.com's blacklist feature

  • HackRepair is enalbed, then .htaccess is modified to add user agents and referrer rules for blocking (Forbidden) access.

Local Brute Force Protection :: Ban hosts and users with invalid login attempts 404 Detection :: locks out someone that gets too many 404 pages Global Settings :: Lockout White List by IP

Network Brute Force Protection :: enabled by default WordPress Tweaks : enabled by default

Away mode :: Disable access to wp Dashboard on a schedule

Scan site to report vulnerabilities and try to fix them Ban bad user agents, bots and other hosts Enforce strong passwords for all accounts Force SSL for admin pages and any pages Turn off file editing from wp-admin UI Detect and block attacks to filesystem and database Change URLs of login, admin Turn off the ability to login for a given time period Remove theme, plugin and core update notifications from users who do not have permission to update them Remove Windows Live Write header info Remove RSD header info Rename admin account Change ID on the user with ID 1 Change database table prefix Change wp-content path Remove login error messages

  1. Release

    3.6.5 6.9.2 > 7.0.1

3.30.24.3. exploit-scanner wp:plugin:exploit-scanner

Search the files on your website, and the posts and comments tables of your database for anything suspicious. It also examines your list of active plugins for unusual filenames.

You need to have wp-content/plugins/exployt-scanner/hashes-4.9.6.php to match the wp version in order for the file checking to work correct https://github.com/philipjohn/exploit-scanner-hashes

3.30.25. Authentication

3.30.25.1. WordPress OAuth Server
  • Premium without trial https://wp-oauth.com
  • Installation
    • Plugin > Settings > General
      • OAuth Server Enabled
    • Plugin > Settings > Advanced Config
      Authorization Code
      HTTP redirects and WP login form when authenticating
      Allow Implicit
      Enable

    • Plugin > create a client

      • Redirect URI
      • Get the client ID
      Open a InCognito Window and test the Auth
      http://mylocalwpauthserver:8000/oauth/authorize?client_id={xxx}&response_type=token it will redirect to login page, after login, it will redirect to the Redirect URI previously set
      Get the Bearer token from the Redirect URI
      https://redirected.io/#access_token={xxx}&expires_in={yyy}&token_type=Bearer&other

3.30.26. Email

3.30.26.1. sendgrid-email-delivery-simplified
  • Create API key with Mail Send: Full permission only
  • Create Sender Authentication: Domain and Link Branding
  • On WP, use Send Mail with API
  • Specify category yourwebsite.com
  • Hard set API key in code otherwise the setting page keeps refreshing as browser form auto-fill kicks in.

wp-config.php 

define('SENDGRID_API_KEY', 'your.api.key');

3.30.27. Menu, Nav, Breadcrumb

3.30.27.1. menu-image
3.30.27.2. megamenu megamenu-pro wp:plugin:megamenu
Install free version first
https://wordpress.org/plugins/megamenu/ then install megamenu-pro

https://www.megamenu.com/documentation/

To enable Max Mega Menu for a menu, put this menu into a theme location.

  1. Search box

    Create a menu with custom link, go to Mega Menu > Replacements > Type Search box

    • Icon color
    • Don't need to modify Styling, everything is under Replacements
  2. CSS
    1. Desktop menu
      /* div#mega-menu-wrap-primary */
#{$wrap} {
    /* ul#mega-menu-primary */
    #{$menu} {
    /* Top level menu items for desktop and mobile */
    & > li.mega-menu-item {
        & > a.mega-menu-link {
        }
        /* sub menu of the Top level menu */
        & > ul.mega-sub-menu {
        }
    }
    }
}
    2. Mobile menu
      /** Push menu onto new line **/ 
#{$wrap} { 
    clear: both;

    /* .mega-toggle-* mobile menu */

    /* Evenly distributed on the first column */
    .mega-toggle-blocks-left {
    .mega-toggle-block {
        width:25%;
        text-align:center;
        margin-left:0!important;
        a.mega-icon {
        &:before {
            display:block;
        }
        &:hover, &:focus {
            &:before {
            color:#cd1713;
            }
        }
        }
    }
    }

    /* Hide middle column */
    .mega-toggle-blocks-center {
    display:none!important;
    }
}
  3. Turn off mobile menu

    Set breakpoint to 0

  4. WPML

    WPML > Languages > Make themes work multilingual > enable Adjust IDs for multilingual functionality

    Create a menu for each language using WPML

    However, menus of the same theme location can only have one Mega Menu's Menu Theme. e.g. add translation to Mobile Menu under Menu Theme

    • [wpml_if lang='en']MENU[/wpml_if][wpml_if lang='de']Menü[/wpml_if]
    • It's also possible to translate the logo URL [wpml_if lang='en']http://www.google.com[/wpml_if][wpml_if lang='de']http://www.google.de[/wpml_if]
  5. TS: Menu height zero

    Usually it's caused by the first menu item is set to align left and the other items set to align right.

    Set the default alignment in Mega Menu > Menu Themes > Menu Bar > Menu Items Align to left and set the menu item to Default in Appearance > Menus > Menu Structure > Mega Menu > Settings > Menu Item Align > Default (change from align left)

3.30.27.3. breadcrumb-navxt

After installing and activating the plugin, to get breadcrumb trails to display either use the included widget, or call the breadcrumb trail in your theme (or child theme). See the Calling the Breadcrumb Trail article for more information on calling the breadcrumb trail.

  1. Release

    5.0.1 6.0.4

3.30.28. Slider, Lightbox

3.30.28.1. responsive-lightbox

https://dfactory.eu/docs/responsive-lightbox/manual-usage/ By default, RL applies effects to all published images, videos and galleries.

Settings > Responsive Lightbox > General settings

  • set Selector and enable lightbox for WordPress image links. This is the manual way.
  • (default) Enable lightbox for WordPress image links will apply to all published images
  • set Display single post images as a gallery. This is the manual way to create gallery
  • (default) Add lightbox to WordPress image galleries by default.
  • (default) Add lightbox to YouTube and Vimeo video links by default.
<!-- Manual on single image -->
<a href="image-link" data-rel="lightbox"><img class ="some-text-here" src="image-link" /></a>

<!-- Manual on single gallery -->
<a href="image-link" data-rel="lightbox-gallery-182"><img class ="some-text-here" src="image-link" /></a>
3.30.28.2. Slider Revolution, revslider wp:plugin:revslider
  • On Envato

  • Display in template 

    putRevSlider("home");
putRevSlider("home","homepage"); // only display if current page is homepage
putRevSlider("home","2,10"); // only display if current page has an ID
  • Shortcode: [rev_slider home]
  • Add Revolution Slider widget to the desired sidebar
3.30.28.3. LayerSlider Responsive WordPress Slider Plugin wp:plugin:LayerSlider
3.30.28.4. MetaSlider wp:plugin:metaslider

3.30.29. Widgets

3.30.29.1. widget-logic
3.30.29.2. Mini Loops wp:plugin:mini-loops
Old
Miniloops
(no term)
Used inside shortcode [miniloop]
image, ml_image

attributes

[image from="thumb"]
get featured image. from can also be and comma-separated
attached
attached image
customfield
(no term)
first
  • [image class="myclass"]
  • [image width="100"]
  • [image height="100"
  • ">0 to scale
  • [image fallback="/wp-content/upload/a.jpg"]
  • ">clear to not save to db and always crop/scale image on the fly
3.30.29.3. vertical-marquee-plugin

In your WordPress administrator section go to Settings menu and select Vertical marquee plugin menu to configure this plugin.

Scroll Amount : This is used to organize the speed of the scrolling, only integer allowed and higher value provides a faster speed Scroll Delay : This is used to reduce the speed of the scrolling, only integer allowed.

Drag and drop the widget : Go to widget menu and drag and drop the Vertical marquee widget to your sidebar location.

Short code for pages and posts : Use the below short code in the pages and posts.

[vertical-marguee setting="1" group="group1"]

Add directly in the theme : Add the below PHP code in your theme PHP file, for example if you want to add this slider in your website footer, just activate the plugin and add this code in footer.php file.

From version 1.0 to 4.0

<?php verticalmarquee(); ?>

From version 4.0 onwards

<?php vmarquee( $setting="1", $group="widget" ); ?>

3.30.29.4. vertical-scroll-recent-post
3.30.29.5. black-studio-tinymce-widget
  • Add a "Visual Editor" widget like WYSIWYG TinyMCE switching between Visual and HTML mode
  • Since WP Core 4.8, it has Visual text widget but it's minimal
3.30.29.6. recent-posts-widget-extended
3.30.29.7. widget-css-classes

3.30.30. Ecommerce

3.30.30.1. WP Job Manager wp:plugin:wp-job-manager
  1. Basic
    • Core plugin is free but add-ons are not
    • New user role employer
    • single.php or single-job_listing.php
    • content-widget-job_listing.php
    • Shortcodes
      submit_job_form
      new page out of wp-admin for employers to post new jobs
      job_dashboard
      new page out of wp-admin for employers to manager job listings
      jobs
      new page for visitors to browse, search and filter job listings
    • Setting
      • General
        • Enable Usage Tracking
    1. Add-ons

      Add-on bundle includes several add-ons for a cheaper price

      • $125/yr for one site and $250/yr for unlimited sites. It includes
      • Resume Manager
      • Job Alerts
      • Indeed Integration
      • Job Tags
      • WC Paid Listings
      • Simple Paid Listings
      • Application Deadline
      • Bookmarks
      • Applications
      • Embeddable Job Widget
      • ZipRecruiter
      1. WC Paid Listings
3.30.30.2. Easy Digital Downloads wp:plugin:easy-digital-downloads
3.30.30.3. woocommerce wp:plugin:woocommerce
  1. Basics
  2. Payment Gateways
  3. hooks
    1. Add a custom field to Order :: woocommerce_shop_order_search_fields, manage_shop_order_posts_custom_column

      Add search on custom field 

      add_filter( 'woocommerce_shop_order_search_fields', 'lili_wc_shop_order_search_field_myfield' );
function lili_wc_shop_order_search_field_myfield( $search_fields ) {
  $search_fields[] = '_myfield'; // You can add a WooCommerce builtin order field without creating a field 
  // start with _
  return $search_fields;
}

      Add a column to the 3rd position 

      function ra_order_tax_receipt_columns($columns) {
    $columnsF = array_slice($columns, 0, 2);
    $columnL = array_slice($columns, 2);
    return array_merge($columnsF, array('myfield' => __('My Field Name')), $columnL);
}
add_filter('manage_edit-shop_order_columns' , 'ra_order_tax_receipt_columns', 2);

add_action( 'manage_shop_order_posts_custom_column' , 'lili_wc_shop_order_search_field_myfield_content', 10, 2 );
function lili_wc_shop_order_search_field_myfield_content( $column ) {
    global $post, $woocommerce, $the_order;
    $order_id = $the_order->id;
    switch ( $column ) {
     case 'myfield' :
   $order_items = $the_order->get_items();
   $myVarOne = get_post_meta($order_id, '_myfield', true );
   echo $myVarOne;
   break;
    }
}
    2. Page/Template hooks

      https://businessbloomer.com/woocommerce-visual-hook-guide-archiveshopcat-page/

  4. Product
    $_pf = new WC_Product_Factory();
$_p  = $_pf->get_product( 123 );

echo apply_filters(
  'woocommerce_loop_add_to_cart_link',
  sprintf(
    '<a href="%s" rel="nofollow" data-product_id="%s" data-product_sku="%s" class="button product_type_simple add_to_cart_button ajax_add_to_cart">%s %s</a>',
    esc_url( $_p->add_to_cart_url() ),
    esc_attr( $_p->get_id() ),
    esc_attr( $_p->get_sku() ),
    implode( ' ', array_filter( [
      'button',
      'product_type_' . $_p->product_type,
      $_p->is_purchasable() && $_p->is_in_stock() ? 'add_to_cart_button' : '',
      $_p->supports( 'ajax_add_to_cart' ) ? 'ajax_add_to_cart' : '',
    ] ) ),
    esc_attr( $_p->product_type ),
    $_p->get_price_html(),
    esc_attr( isset( $class ) ? $class : 'button' ),
    esc_html( $_p->add_to_cart_text() )
  ),
  $_p );

    For each product edit page, change Advanced > Menu order and then use menu_order in query: 

    $args = array( 'post_type' => 'product', 'product_cat' => $prodCat, "orderby"=>"menu_order", "order"=>"ASC");
$loop = new WP_Query( $args );
  5. Template Structure

    https://docs.woocommerce.com/document/template-structure/

    wp-content/plugins/woocommerce/templates/emails/admin-new-order.php => wp-content/themes/yourtheme/woocommerce/emails/admin-new-order.php wp-content/plugins/woocommerce/templates/content-single-product.php => wp-content/themes/yourtheme/woocommerce/content-single-product.php

  6. Test Credit Cards

    Turn on test mode for each Payment Gateway. For 38.39.2, WooCommerce > Settings > Payments > Moneris - Credit Card > Manage > Environment > Sandbox

  7. Uninstall and remove data

    In order to remove data in db when removing the plugin, add this first then deactivate and delete the plugin: 

    define( 'WC_REMOVE_ALL_DATA', true);
/* That's all, stop editing! Happy blogging. */
  8. extension: Subscription

    Make recurring payment and subscription products possible. Docs: https://docs.woocommerce.com/document/subscriptions/ https://woocommerce.com/products/woocommerce-subscriptions/

  9. extension: Name Your Price

    Set custom price by buyer. Compatible with Subscription extension.

  10. extension: WooCommerce Sequential Order Numbers Pro

    The post id of order is still not sequential but global $the_order->id is.

3.30.30.4. woo-order-export-lite
3.30.30.5. woocommerce-pdf-invoices-packing-slips

Doc: http://docs.wpovernight.com/woocommerce-pdf-invoices-packing-slips/

  1. Template hooks

    http://docs.wpovernight.com/woocommerce-pdf-invoices-packing-slips/pdf-template-action-hooks/ http://hookr.io/plugins/woocommerce-pdf-invoices-packing-slips/

    Simple template wp-content/plugins/woocommerce-pdf-invoices-packing-slips/templates/Simple/ html-document-wrapper.php invoice.php

    1. action wpo_wcpdf_before_document

      $wpo_wcpdf_export_template_type :: The wpo wcpdf export template type. $wpo_wcpdf_export_order :: The wpo wcpdf export order. 

      function action_wpo_wcpdf_before_document( $wpo_wcpdf_export_template_type, $wpo_wcpdf_export_order ) { 
    // make action magic happen here...
}; 

// add the action 
add_action( 'wpo_wcpdf_before_document', 'action_wpo_wcpdf_before_document', 10, 2 );
    2. action wpo_wcpdf_after_document_label
    3. action wpo_wcpdf_before_billing_address

3.30.31. Theme

3.30.31.1. wpfront-scroll-top

3.30.32. Page Builder

3.30.32.1. Elementor
3.30.32.2. Beaver Builder
3.30.32.4. Thrive Architect
3.30.32.5. SiteOrigin Page Builder
  • https://siteorigin.com/page-builder/
  • Free to start
    • 20 themes that support Page Builder plugin (separate install)
    • 16 SiteOrigin Site Packs: separate install, each is a full website
    • Can work with free version of the following plugins
      Widgets Bundle
      about 23 widgets
      (no term)
      SiteOrigin CSS
      (no term)
      SiteOrigin Installer
  • Premium version
    • Addons
    • Enhanced Page Builder
    • Extra widgets in plugin Widgets Bundle
    • Theme Enhancements
3.30.32.6. WPBakery (old name: Visual Composer)

Config 

# BEGIN WordPress
# ...
# END WordPress

SubstituteMaxLineLength 10m

# if still doesn't work
#WPBakery Timeout fixer - .htaccess
SubstituteMaxLineLength 10M
LimitRequestBody 9999999
  1. Media Grid (Old)

    https://codecanyon.net/item/media-grid-wordpress-responsive-portfolio/2218545

3.30.33. jetpack

  • Build wp with themes
  • unlimited image CDN from Photon
  • lazy loading images
  • brute force attack protection
  • downtime monitoring
  • automated social media posting

3.31. Debug, WP_Error

https://codex.wordpress.org/Function_Reference/WP_Error Function may return WP_Error object. Catch error 

$post = wp_insert_post([]);
if ( is_wp_error($post) ) {
    echo $post->get_error_message();
}

3.31.1. Show error wp:show error

if ( ! defined( 'WP_DEBUG' ) ) {

    if ( php_sapi_name() == 'cli' ||
         ( isset( $_SERVER['PANTHEON_ENVIRONMENT'] ) && $_ENV['PANTHEON_ENVIRONMENT'] === 'live' )
    ) {
        // Force to turn off debug for Live environment and WP-CLI
        define( 'WP_DEBUG', false );
    } elseif ( 1 == 1 ) {
        // toggle debug
        define( 'WP_DEBUG', true );
        define( 'WP_DEBUG_DISPLAY', true ); // set to false to not display error on HTML pages
        define( 'SAVEQUERIES', true ); // enable global $wpdb->queries to analyze queries
        @error_reporting( E_ALL );
        @ini_set( 'log_errors', true );
        @ini_set( 'log_errors_max_len', '0' );
        @ini_set( 'display_errors', 1 ); // 'stderr' can be set

        // define( 'WP_DEBUG_LOG', true ); // /wp-content/debug.log useful to debug AJAX or wp-cron run
        // define( 'CONCATENATE_SCRIPTS', false ); // by default, for Admin area, WP loads one JavaScript. If you have one error in script, it will fail
        // set to false that JavaScript files are loaded separately for Admin area.
    } else {
        define( 'WP_DEBUG', false );
    }

}

Or anywhere 

error_reporting(E_ALL);
ini_set('display_errors', 1);

Write to file wp-content/debug.log after define( 'WP_DEBUG_LOG' , true ); 

error_log('Hello lili');
error_log(print_r($abc, 1));

3.31.2. Debug WP_Query

About SAVEQUERIES 

if ( current_user_can( 'administrator' ) ) {
    global $wpdb;
    echo "<pre>";
    print_r( $wpdb->queries );
    echo "</pre>";
}

$query = new WP_Query($args);

// debug: actual SQL raw query that is run
echo "<pre>";
print_r($query->request);
// to see what query vars are passed to wp_query
// print_r($query->query_vars);
echo "</pre>";

$args = array(
  'post_type' => array('news'),
    // 'tax_query' => array(...),
    // ...
    'lili' => 'lili'
);

// inside class-wp-query.php function get_posts()
if (isset($q['lili'])) {
    //var_dump($where);
}

3.31.3. Which template? wp:global:template

3.31.4. All Image Sizes 

add_action( 'wp_footer', 'debug_all_image_sizes' );
function debug_all_image_sizes() {
    print '<h1>All Image Sizes</h1>';
    print '<pre>';
    global $_wp_additional_image_sizes;
    print_r( $_wp_additional_image_sizes );
    // [ 'image-size-name' => [ 'width' => 300, 'height' => 250, 'crop' => 1 ] ]
    // crop is empty means the image size is not croppable
    print '</pre>';
}
function debug_template_file() {
    print '<h1>All Image Sizes</h1>';
    print '<pre>';
    global $template;
    print_r( $template );
    print '</pre>'; 
}

3.31.5. Filters 

if ( WP_DEBUG && array_intersect( array( 'administrator' ), wp_get_current_user()->roles ) ) {
    add_action( 'wp_footer', 'debug_genesis_attr_filters' );
}

function debug_genesis_attr_filters() {
    global $wp_filter; // current_filter() might be a better way to do this
    $genesis_attr_filters = array();

    $h1  = '<h1>Current Page Genesis Attribute Filters</h1>';
    $out = '';
    $ul  = '<ul>';

    foreach ( $wp_filter as $key => $val ) {
        if ( false !== strpos( $key, 'genesis_attr' ) ) {
            $genesis_attr_filters[ $key ][] = var_export( $val, true );
        }
    }
    foreach ( $genesis_attr_filters as $name => $attr_vals ) {
        $out .= "<h2 id=$name>$name</h2><pre>" . implode( "\n\n", $attr_vals ) . '</pre>';
        $ul  .= "<li><a href='#$name'>$name</a></li>";
    }
    print "$h1$ul</ul>$out";
}

global $wp_filter;
print '<pre>';
// all filters
// print_r($wp_filter);

// a specific filter
// print_r($wp_filter['posts_where']);

// More advanced: find file name and line number of the callback function
// array(
//   filter 1
//   array( id, priority, function, accepted_args, file, line )
// )
print_r(list_hooks('posts_where'));
print '</pre>';

// https://stackoverflow.com/a/26680808/2196360
function list_hooks( $hook = '' ) {
    global $wp_filter;

    if ( isset( $wp_filter[ $hook ]->callbacks ) ) {
        array_walk( $wp_filter[ $hook ]->callbacks, function ( $callbacks, $priority ) use ( &$hooks ) {
            foreach ( $callbacks as $id => $callback ) {
                $hooks[] = array_merge( [ 'id' => $id, 'priority' => $priority ], $callback );
            }
        } );
    } else {
        return [];
    }

    foreach ( $hooks as &$item ) {
        // skip if callback does not exist
        if ( ! is_callable( $item['function'] ) ) {
            continue;
        }

        // function name as string or static class method eg. 'Foo::Bar'
        if ( is_string( $item['function'] ) ) {
            $ref          = strpos( $item['function'], '::' ) ? new ReflectionClass( strstr( $item['function'], '::', true ) ) : new ReflectionFunction( $item['function'] );
            $item['file'] = $ref->getFileName();
            $item['line'] = get_class( $ref ) == 'ReflectionFunction'
                ? $ref->getStartLine()
                : $ref->getMethod( substr( $item['function'], strpos( $item['function'], '::' ) + 2 ) )->getStartLine();

            // array( object, method ), array( string object, method ), array( string object, string 'parent::method' )
        } elseif ( is_array( $item['function'] ) ) {

            $ref = new ReflectionClass( $item['function'][0] );

            // $item['function'][0] is a reference to existing object
            $item['function'] = array(
                is_object( $item['function'][0] ) ? get_class( $item['function'][0] ) : $item['function'][0],
                $item['function'][1]
            );
            $item['file']     = $ref->getFileName();
            $item['line']     = strpos( $item['function'][1], '::' )
                ? $ref->getParentClass()->getMethod( substr( $item['function'][1], strpos( $item['function'][1], '::' ) + 2 ) )->getStartLine()
                : $ref->getMethod( $item['function'][1] )->getStartLine();

            // closures
        } elseif ( is_callable( $item['function'] ) ) {
            $ref              = new ReflectionFunction( $item['function'] );
            $item['function'] = get_class( $item['function'] );
            $item['file']     = $ref->getFileName();
            $item['line']     = $ref->getStartLine();

        }
    }

    return $hooks;
}

3.32. Installation Language

Download additional language packs from here Upload thme to wp-content/languages Then change the language in each User Profile.

3.33. File Permissions 

# set permission 644 on files
find . -type f -exec chmod 644 {} +
# set permissions 755 for directories
find . -type d -exec chmod 755 {} +
chmod 660 wp-config.php

Run this to check which files are not 644 and which directories are not 755 wp:check file permissions 

find . -type d -not -perm 755 -o -type f -not -perm 644
find . -type d -not -perm 755 -not -path "./.git/*" -o -type f -not -perm 644 -not -path "./.git/*"

Multiple users upload as www-data linux:permission:users

3.34. SSL

// Force HTTPS logins and administration
define('FORCE_SSL_ADMIN', true);

// The above also means this below
//define('FORCE_SSL_LOGIN', true);

define('WP_HOME','https://'.$_SERVER[HTTP_HOST]);
define('WP_SITEURL','https://'.$_SERVER[HTTP_HOST]);

You can now try to go to wp-admin If W3 Total Cache is installed, Performance > Page Cache > enable Cache SSL (https) requests If some security plugin is installed, make sure no SSL is enforced for any pages because Apache at the end will cover that.

Use ssl:test tools to identify problematic links Fix legacy content in DB https://css-tricks.com/moving-to-https-on-wordpress/ 

-- double quotes
UPDATE wp_posts 
SET    post_content = ( Replace (post_content, 'src="http://', 'src="//') )
WHERE  Instr(post_content, 'jpeg') > 0 
    OR Instr(post_content, 'jpg') > 0 
    OR Instr(post_content, 'gif') > 0 
    OR Instr(post_content, 'png') > 0;

-- single quote
UPDATE wp_posts 
SET   post_content = ( Replace (post_content, "src='http://", "src='//") )
WHERE  Instr(post_content, 'jpeg') > 0 
    OR Instr(post_content, 'jpg') > 0 
    OR Instr(post_content, 'gif') > 0 
    OR Instr(post_content, 'png') > 0;

-- Custom Fields
UPDATE wp_postmeta 
SET meta_value=(REPLACE (meta_value, 'iframe src="http://','iframe src="//'));

Force redirect to https on Apache for all pages :: apache:https To setup nginx as host and apache as proxy nginx:proxy:docker

If WordPress is behind Nginx proxy server or load balancer, the above is not enough because is_ssl function will not work Inspired by plugin ssl-insecure-content-fixer wp-config.php 

if (isset($_SERVER['HTTP_X_FORWARDED_PROTO']) && strtolower($_SERVER['HTTP_X_FORWARDED_PROTO']) === 'https') {
  $_SERVER['HTTPS'] = 'on';
}

3.35. REST API wp:rest

https://deliciousbrains.com/wordpress-rest-api-vs-custom-request-handlers/

3.35.1. Methods (Also called operations, verbs, routes)

/tickets
is called endpoint
GET /tickets
Retrieves a list of tickets
OPTIONS /tickets
see any parameters and fields can be used for endpoint tickets
GET /tickets/12
Retrieves a specific ticket
POST /tickets
Creates a new ticket
PUT /tickets/12
Updates ticket #12
PATCH /tickets/12
Partially updates ticket #12
DELETE /tickets/12
Deletes ticket #12

3.35.2. Status Codes

https://restfulapi.net/http-status-codes/

3.35.2.1. 1xx Transfer protocol-level info
100
An interim response. Indicates the client that the initial part of the request has been received and has not yet been rejected by the server. The client should continue by sending the remaminder of the request, or if the request has already been completed, ignore this response. The server MUST send a final response after the request has been completed
102
Processing (WebDav). Indicates the server has received and is processing the request, but no response is available yet
3.35.2.2. 2xx Success
200
OK. Should have a response body
201
Created. Should return URI of the newly created resource
202
Accepted. Used for actions that take a long while to process. Indicates the request has been accepted for processing but it's not completed. The request might or might not be eventually acted upon, or even maybe disallowed when processing occurs. Should return the request's current status and either a pointer to a status monitor (job queue location) or some estimate of when the user can expect the request to be fulfilled
203
Non-authoritative information. The returned metainformation in the entity-header is not the definitive set as available from the origin server, but is gathered from a local or a third-party copy. The set presented MAY be a subset or superset of the original version
204
No content
  • Usually it's sent out in response to PUT, POST or DELETE request when REST API declines to send back any status message or representation in the response body
  • In response to GET request to indicate that the requested resource exists but has no state representation to include in the body
  • Must not include a message-body and thus is always terminated by the first empty line after the header fields
3.35.2.3. 3xx Redirection - The client must take some additional action in order to complete their request
300
multiple choices. The request has more than one possible response. The user-agent or user should choose one of them
301
Moved permanently. The URL of the requested resource has been changed permanently. The new URL is given by the Location header field in the resposne. This response is cachable unless indicated otherwise
302
Found. The URL of the requested resource has been changed temporarily. The new URL is given by the Location field in the response. This response is only cacheable if indicated by a Cache-Control or Expires header field
303
See other. The response can be found under a different URI and SHOULD be retrieved using a GET method on that resource
304
Not modified. The client that the response has not been modified, so the client can continue to use the same cached version of the response
307
Temporary redirect. The REST API is not going to process the client's request. Instead, the client should resubmit the request to the URI specified by the response message's Location header. However, future requests should still use the original URI. REST API can use this status code to assign a temporary URI to the client's requested resource. E.g. 307 response can be used to shift a client request over to another host.
  • The temporary URI SHOULD be given by the Location field in the response. Unless the request method was HEAD, the entity of the response SHOULD contain a short hypertext note with a hyperlink to the new URI(s). If the 307 status code is received in response to a request other than GET or HEAD, the user agent MUST NOT automatically redirect the request unless it can be confirmed by the user, since this might change the conditions under which the request was issued
3.35.2.4. 4xx Client error
400
Bad request. General purpose when no other 4xx code is appropriate. The client SHOULD NOT repeat the request without modifications
401
Unauthorized. MUST include a WWW-Authenticate header field containing a challenge applicable to the requested resource. The client MAY repeat the request which a suitable Authorization header field. If the request already included Authorization credentials, then the 401 response indicates that authorization has been refused for those credentials. If the 401 response contains the same challenge as the prior response, and the user agent has already attempted authentication at least once, then the user SHOULD be presented the entity that was given in the response, since that entity might include relevant diagnostic information
403
Forbidden. The client request is formed correctly, but the REST API refuses to honor it e.g. the user does not have the necessary permissions for the resource. It's not a case of insufficient client credentials, that would be 401
  • Authentication will not help, and the request SHOULD NOT be repeated
405
Method not allowed. Must include the Allow header, which lists the HTTP methods that the resource supports e.g. Allow: GET, POST
406
Not acceptable. A client request for data formatted as application/xml will receive 406 if the API is only willing to format data as application/json. API is not able to generate any of the client's preferred media types as indicated by the Accept request header. A user agent should temporarily stop receipt of more data and query the user for a decision on further actions
412
Precondition failed. The client specified one or more preconditions in its request headers, effectively telling the REST API to carry out its request only if certain conditions were met
415
Unsupported media type. API is not able to process tthe client's supplied media type, as indicated by the Content-Type request header
3.35.2.5. 5xx Server error
500
Generic. e.g. when the server raises an exception
501
Not implemented. The server either does not recognize the request method, or it cannot fulfill the request. Usually implies future availability

3.35.3. Global parameters

https://developer.wordpress.org/rest-api/using-the-rest-api/global-parameters/

3.35.3.1. _jsonp
<script>
function receiveData( data ) {
  // Do something with the data here.
  // For demonstration purposes, we'll simply log it.
  console.log( data );
}
</script>
<script src="https://demo.wp-api.org/wp-json/?_jsonp=receiveData"></script>
3.35.3.2. _method

This is for server that couldn't fire a DELETE HTTP request A POST to /wp-json/wp/v2/posts/42?_method=DELETE would be translated to a DELETE to the wp/v2/posts/42 route.

Similarly, the following POST request would become a DELETE: 

POST /wp-json/wp/v2/posts/42 HTTP/1.1
Host: example.com
X-HTTP-Method-Override: DELETE
3.35.3.3. _envelope

Some servers, clients, and proxies do not support accessing the full response data. The API supports passing an _envelope parameter, which sends all response data in the body, including headers and status code.

GET to wp/v2/users/me: 

HTTP/1.1 302 Found
Location: http://example.com/wp-json/wp/v2/users/42

{
  "id": 42,
  ...
}

GET to wp/v2/users/me?_envelope: 

HTTP/1.1 200 OK

{
  "status": 302,
  "headers": {
    "Location": "http://example.com/wp-json/wp/v2/users/42"
  },
  "body": {
    "id": 42
  }
}
3.35.3.4. _embed

3.35.4. Pagination and Ordering

  • ?page=
  • default 10, possible from 1 to 100
  • ?offset=
  • asc desc
  • date author relevance id include modified parent relevance slug include_slugs title

3.35.5. Response Header fields

X-WP-TOTAL
total number of records
(no term)
X-WP-TotalPages

3.35.6. Endpoints

To see all endoints
Method:OPTIONS http://mysite.io/wp-json/wp-v2/
3.35.6.1. Plugin rest-filter

Use https://github.com/wp-api/rest-filter to build more complex queries

Term slug of taxonomy news-category wp-json/wp/v2/news?per_page=5&filter[news-category]=news-abc

3.35.6.2. Custom Post Types and Taxonomies
  • Make sure both post type and taxonomy has show_in_rest=true in wp:f:register_post_type~ and wp:f:register_taxonomy
    rest_base
    change the base url of REST API route
    rest_controller_class
    Default is WP_REST_Posts_Controller
  • Core cannot query term slug
  • wp-json/wp/v2/news?per_page=5&news-category=1234
  • Multiple terms
    • wp-json/wp/v2/news?per_page=5&news-category=1234,4567
  • Multiple terms in multiple taxonomies
    • wp-json/wp/v2/news?per_page=5&news-category=1234,4567&abc-category=789,123
3.35.6.3. posts
3.35.6.4. Custom endpoint
  1. Simple way
    // http://example.com/wp-json/myplugin/v1/author/(?P<id>\d+)
add_action( 'rest_api_init', function () {
    // version 1
    register_rest_route( 'myplugin/v1', '/author/(?P<id>\d+)', array(
        'methods'  => 'GET',
        'callback' => 'my_awesome_func',
        /*' optional
        permission_callback' => function () {
            return current_user_can( 'edit_others_posts' );
        },*/
        'args'     => array(
            'id' => array(
                // 'default' => 'some default value',
                // 'sanitize_callback' => function() {},
                // 'sanitize_callback' => 'absint',
                'validate_callback' => function ( $param, $request, $key ) {
                    return is_numeric( $param );
                }
            ),
        ),
    ) );
} );

/**
 * Grab latest post title by an author!
 *
 * @param array $data Options for the function.
 *
 * @return string|null Post title for the latest,
 * or null if none.
 */
function my_awesome_func( WP_REST_Request $data ) {

    $posts = get_posts( array(
        'author' => $data['id'],
    ) );

    if ( empty( $posts ) ) {
        return new WP_Error( 'awesome_no_author', 'Invalid author', array( 'status' => 404 ) );
    }

    $data = $posts[0]->post_title;
    // $data = array( 'some', 'response', 'data' );

    // Direct return response
    // turn data into a WP_REST_Response object
    // return rest_ensure_response($data);

    // Or return a WP_REST_Response object

    // Create the response object
    $response = new WP_REST_Response( $data );

    // Add a custom status code
    $response->set_status( 201 );

    // Add a custom header
    //$response->header( 'Location', 'http://example.com/' );

    return $response;
}
  2. Using controller
    add_action( 'rest_api_init', 'init_wp_rest_multiple_posttype_endpoint' );

function init_wp_rest_multiple_posttype_endpoint() {
    if ( ! class_exists( 'WP_REST_Multiple_PostType_Controller' ) ) {
        require_once dirname( __FILE__ ) . '/lib/endpoints/class-wp-rest-multiple-posttype-controller.php';
    }
    $controller = new WP_REST_Multiple_PostType_Controller();
    $controller->register_routes();
}

    theme_path/lib/endpoints/class-wp-rest-multiple-posttype-controller.php

3.35.6.5. Batch

https://developer.wordpress.org/rest-api/requests/

  • Batch run multiple REST API GET requets
  • One HTTP GET request but return multiple responses as different key in JSON response
// example.com/wp-json/my-namespace/v1/batch?requests[0][method]=GET&requests[0][route]=/wp/v2/news&requests[0][params][per_page]=6&requests[0][params][orderby]=rand&requests[0][params][filter][center]=game-changer

// /wp-json/wp/v2/news?per_page=6&filter[centre]=game-changer&orderby=rand

// response
// { 'GET /wp/v2/news': [ response for that REST API v2 request ]}

// Register our mock batch endpoint.
function prefix_register_batch_route() {
    register_rest_route( 'my-namespace/v1', '/batch', array(
        // Supported methods for this endpoint. WP_REST_Server::READABLE translates to GET.
        'methods' => WP_REST_Server::READABLE,
        // Register the callback for the endpoint.
        'callback' => 'prefix_do_batch_request',
        // Register args for the batch endpoint.
        'args' => prefix_batch_request_parameters(),
    ) );
}

add_action( 'rest_api_init', 'prefix_register_batch_route' );

/**
 * Our registered endpoint callback. Notice how we are passing in $request as an argument.
 * By default, the WP_REST_Server will pass in the matched request object to our callback.
 *
 * @param WP_REST_Request $request The current matched request object.
 */
function prefix_do_batch_request( $request ) {
    // Here we initialize the array that will hold our response data.
    $data = array();
    $data = prefix_handle_batch_requests( $request['requests'] );
    return $data;
}

/**
 * This handles the building of the response for the batch requests we make.
 *
 * @param array $requests An array of data to build WP_REST_Request objects from.
 * @return WP_REST_Response A collection of response data for batch endpoints.
 */
function prefix_handle_batch_requests( $requests ) {
    $data = array();

    // Foreach request specified in the requests param run the endpoint.
    foreach ( $requests as $request_params ) {
        $response = prefix_handle_request( $request_params );
        $key = $request_params['method'] . ' ' . $request_params['route'];
        $data[ $key ] = prefix_prepare_for_collection( $response );
    }

    return rest_ensure_response( $data );
}

/**
 * This handles the building of the response for the batch requests we make.
 *
 * @param array $request_params Data to build a WP_REST_Request object from.
 * @return WP_REST_Response Response data for the request.
 */
function prefix_handle_request( $request_params ) {
    $request = new WP_REST_Request( $request_params['method'], $request_params['route'] );

    // Add specified request parameters into the request.
    if ( isset( $request_params['params'] ) ) {
        foreach ( $request_params['params'] as $param_name => $param_value ) {
            $request->set_param( $param_name, $param_value );
        }
    }
    $response = rest_do_request( $request );
    return $response;
}

/**
 * Prepare a response for inserting into a collection of responses.
 *
 * This is lifted from WP_REST_Controller class in the WP REST API v2 plugin.
 *
 * @param WP_REST_Response $response Response object.
 * @return array Response data, ready for insertion into collection data.
 */
function prefix_prepare_for_collection( $response ) {
    if ( ! ( $response instanceof WP_REST_Response ) ) {
        return $response;
    }

    $data = (array) $response->get_data();
    $server = rest_get_server();

    if ( method_exists( $server, 'get_compact_response_links' ) ) {
        $links = call_user_func( array( $server, 'get_compact_response_links' ), $response );
    } else {
        $links = call_user_func( array( $server, 'get_response_links' ), $response );
    }

    if ( ! empty( $links ) ) {
        $data['_links'] = $links;
    }

    return $data;
}

/**
 * Returns the JSON schema data for our registered parameters.
 *
 * @return array $params A PHP representation of JSON Schema data.
 */
function prefix_batch_request_parameters() {
    $params = array();

    $params['requests'] = array(
        'description'        => esc_html__( 'An array of request objects arguments that can be built into WP_REST_Request instances.', 'my-text-domain' ),
        'type'               => 'array',
        'required'           => true,
        'validate_callback'  => 'prefix_validate_requests',
        'items'              => array(
            array(
                'type' => 'object',
                'properties' => array(
                    'method' => array(
                        'description' => esc_html__( 'HTTP Method of the desired request.', 'my-text-domain' ),
                        'type'        => 'string',
                        'required'    => true,
                        'enum'        => array(
                            'GET',
                            'POST',
                            'PUT',
                            'DELETE',
                            'OPTIONS',
                        ),
                    ),
                    'route' => array(
                        'description' => esc_html__( 'Desired route for the request.', 'my-text-domain' ),
                        'required'    => true,
                        'type'        => 'string',
                        'format'      => 'uri',
                    ),
                    'params' => array(
                        'description' => esc_html__( 'Key value pairs of desired request parameters.', 'my-text-domain' ),
                        'type' => 'object',
                    ),
                ),
            ),
        ),
    );

    return $params;
}

function prefix_validate_requests( $requests, $request, $param_key ) {
    // If requests isn't an array of requests then we don't process the batch.
    if ( ! is_array( $requests ) ) {
        return new WP_Error( 'rest_invald_param', esc_html__( 'The requests parameter must be an array of requests.' ), array( 'status' => 400 ) );
    }

    foreach ( $requests as $request ) {
        // If the method or route is not set then we do not run the requests.
        if ( ! isset( $request['method'] ) || ! isset( $request['route'] ) ) {
            return new WP_Error( 'rest_invald_param', esc_html__( 'You must specify the method and route for each request.' ), array( 'status' => 400 ) );
        }

        if ( isset( $request['params'] ) && ! is_array( $request['params'] ) ) {
            return new WP_Error( 'rest_invald_param', esc_html__( 'You must specify the params for each request as an array of named key value pairs.' ), array( 'status' => 400 ) );
        }
    }

    // This is a black listing approach to data validation.
    return true;
}

3.35.7. Authentication

Require auth for all requests 

add_filter( 'rest_authentication_errors', function( $result ) {
    if ( ! empty( $result ) ) {
    return $result;
    }
    if ( ! is_user_logged_in() ) {
    return new WP_Error( 'rest_not_logged_in', 'You are not currently logged in.', array( 'status' => 401 ) );
    }
    return $result;
});

3.35.8. Current request is REST request?

if ( defined('REST_REQUEST') && REST_REQUEST) {}

3.35.9. New internal request

$request = new WP_REST_Request( 'GET', '/wp/v2/posts' );
// Set one or more request query parameters
$request->set_param( 'per_page', 20 );
$response = rest_do_request( $request );

3.35.10. New external request and read response

$request = wp_remote_get( 'https://a.ca/wp-json/wp/v2/news?per_page=5' );
if ( is_wp_error( $request ) ) {
    return false; // Bail early
}

$body = wp_remote_retrieve_body( $request );

$data = json_decode( $body );

if ( ! empty( $data ) ) {
    echo $data[0]->title->rendered;
}

3.35.11. Cache

I think REST response is object cache. When using with Pantheon Advanced Page Cache and WP Redis plugins, REST response is cached and it's refreshed when necessary after post CRUD.

3.35.12. Hooks

3.35.12.1. rest_api_init
register_rest_field()

Add an extra field in response 

add_action( 'rest_api_init', 'll_rest_api_add_fields' );
function ll_rest_api_add_fields() {
//Add featured image
    register_rest_field(
        array('product'), // Where to add the field (Here, blog posts. Could be an array)
        'featured_image_src', // Name of new field (You can call this anything)
        array(
            'get_callback'    => function( $object, $field_name, $request ) {
                $feat_img_array = wp_get_attachment_image_src(
                    $object['featured_media'], // Image attachment ID
                    'thumbnail',  // Size.  Ex. "thumbnail", "large", "full", etc..
                    true // Whether the image should be treated as an icon.
                );
                return $feat_img_array[0];
    },
            'update_callback' => null,
            'schema'          => null,
        )
    );

    // add some custom fields
    register_rest_field( 'product', 'post_meta_fields', array(
        'get_callback' => function( $post ) {
            $fields = [
                'image'              => get_field( 'product_image' )['url'],
                'product_categories' => (array) wp_get_post_terms( $post['id'], 'product_category' ),
                'company'            => (string) get_field( 'company_name', $post['id'] ),
                'booth'              => (string) get_field( 'booth_number', $post['id'] ),
            ];

            return $fields;
        },
    ) );
}
3.35.12.2. rest_after_insert_{post_type}
  • Default no action is added to the hook
  • Action runs when task post is modified via REST API, Gutenberg or Post editor
add_action( 'rest_after_insert_task', 'taskbook_change_status', 10, 2 );
function taskbook_change_status( $post, $request ) {
    $outcome = get_post_meta( $post->ID, 'taskbook_outcome', true );

    if ( 0 === strlen($outcome) ) {
    update_post_meta( $post->ID, 'task_status', 'In progress' );
    } else {
    update_post_meta( $post->ID, 'task_status', 'Completed' );
    }
}
3.35.12.3. rest_{$this->post_type}_collection_params wp:rest:hook:rest_*_collection_params
  • Register custom collection parameters $_GET['yourparam']
  • Add options to existing parameter e.g. add rand to orderby
  • For post, use rest_post_collection_params
// add rand to orderby for post type news
add_filter( 'rest_news_collection_params', 'my_prefix_add_rest_orderby_params', 10, 1 );
function my_prefix_add_rest_orderby_params( $params ) {
    $params['orderby']['enum'][] = 'rand';
    return $params;
}

3.35.13. Basic Auth

  • https://github.com/WP-API/Basic-Auth

    • Enable WP API to authenticate using Authorization: Basic base64HasedOfUsername:Password. To submit request from client: 

      curl --user admin:password https://example.com/wp-json/

      $args = array(
    'headers' => array(
        'Authorization' => 'Basic ' . base64_encode( $username . ':' . $password ),
    ),
);

3.35.14. Nonce

$.ajax({
    url: WPsettings.root + 'wp/v2/posts/' + WPsettings.current_ID,
    method: 'POST',
    beforeSend: function(xhr) {
    xhr.setRequestHeader( 'X-WP-Nonce', WPsettings.nonce);
    },
    data: {
    'title': newTitle
    }
})

3.35.15. JWT - JSON Web Token

Plugin
https://en-ca.wordpress.org/plugins/jwt-authentication-for-wp-rest-api/

  • Add auth endpoints

    • POST /wp-json/jwt-auth/v1/token with raw JSON in body { "username": "", "password": "" }
    • Token is retrieved. Then in subsequent requests include this header Authorization: Bearer $token

  • Need to add to wp-config.php 

    define('JWT_AUTH_SECRET_KEY', 'your-top-secret-key');
define('JWT_AUTH_CORS_ENABLE', true);

  • Enable HTTP Authroization header, e.g. Apache 

    RewriteEngine on
RewriteCond %{HTTP:Authorization} ^(.*)
RewriteRule ^(.*) - [E=HTTP_AUTHORIZATION:%1]

3.36. SOAP vs RESTful API

SOAP
Simple Object Access Protocol
  • only allows XML
  • Used to expose components of application logic as service rather than data
    • perform operations through a standardized set of messaging patterns

  • Example 1: getMessage request to SOAP server 

    <soap:Envelope
    xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/03/addressing"
    xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"
    xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
    <soap:Header>
        <wsse:Security soap:mustUnderstand="1">
            <wsse:UsernameToken>
                <wsse:Username>%%wsse:Username%%</wsse:Username>
                <wsse:Nonce>%%wsse:Nonce%%</wsse:Nonce>
                <wsu:Created>%%wsu:Created%%</wsu:Created>
                <wsse:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText">%%wsse:Password%%</wsse:Password>
            </wsse:UsernameToken>
        </wsse:Security>
    </soap:Header>
    <soap:Body>
        <getMessage xmlns="http://soap.server.ca">
            <channel>%%linkid%%</channel>
        </getMessage>
    </soap:Body>
</soap:Envelope>
    • Link ID or Channel ID: each channel may have multiple different users
    • Nonce is completely random for each request
    • Password is hashed with time and the real password. Related to wsu:Created
REST
Representational State Transfer
  • Used to access data (named resources)

3.37. Life Cycle

3.37.1. index.php

3.37.2. wp-admin/index.php

  • wp-admin/index.php
    • wp-admin/admin.php
    • wp-admin/includes/dashboard.php
    • wp-admin/admin-header.php
    • wp-admin/admin-footer.php

3.37.3. wp-admin/admin-ajax.php

  • Request wp-admin/admin-ajax.php
    • /wp-load.php
    • /wp-config.php
    • /wp-settings.php (core files, active plugins and themes and the REST API)
    • /wp-admin/admin.php
    • /wp-admin/includes/ajax-actions.php

3.37.4. Request REST API

  • Request REST API

    • index.php

      • wp-blog-header.php (Environment and Template)
      • wp-load.php
        • wp-config.php
        • wp-settings.php (which loads most core files, all active plugins and themes, and the REST API) (doesn't load admin_init, /wp-admin/*.*)

3.38. PHP 7

3.39. Tools

  • wpscans.com

3.40. UC: Simple HTML page with form submit

Create a url path /abc which points to index.html (can't have php files)

3.40.1. /abc/index.html

<script src='https://www.google.com/recaptcha/api.js'></script>

<div id="form-frame" class="form">
  <div class="form-input">
    <div id="err-message"></div>
  </div>
  <div class="form-input">
    <label for="li-fname">Full Name</label>
    <input type="text" name="li-fname" id="li-fname">
  </div>
  <div class="form-input news-input">
    <label for="li-sub-promo">How did you hear about this promotion?</label>
    <select name="li-sub-promo" id="li-sub-promo">
  <option value="Website">Website</option>
  <option value="Radio">Radio</option>
  <option value="TV">TV</option>
  <option value="Referral">Referral</option>
  <option value="Brochure">Brochure</option>
  <option value="Email">Email</option>
  <option value="Newsletter">Newsletter</option>
  <option value="Online Ads">Online Ads</option>
  <option value="Social Media">Social Media</option>
    </select>
  </div>
  <div class="form-input news-input">
    <label for="small-business-lighting">Programs of Interest
  <br>(Check all that apply)</label>
    <div class="li-newsbox">
  <input type="checkbox" name="li-newsbox" id="small-business-lighting">Small
  Business Lighting<br>
  <input type="checkbox" name="li-newsbox"
         id="business-registration-incentives">Business Refrigeration
  Incentives
    </div>
  </div>
  <div class="form-input">
    <label>&nbsp;</label>
    <div class="g-recaptcha" data-sitekey="[[your_google_recaptcha_public_key]]"></div>
    <div id="captcha-verify"></div>
  </div>

  <div class="form-input submit-input" id="form-submit">
    <label>&nbsp;</label>
    <a href="javascript:;" onclick="javascript:submit_form();">Submit</a>
  </div>

</div>
<script type="text/javascript"
    src="/abc/lb/js/app.js"></script>
<script type="text/javascript"
    src="https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>

3.40.2. /abc/lb/js/app.js

function submit_form(){

  var send_msg = document.getElementById('form-submit');
  var pre_send = ('<label>&nbsp;</label><a href="javascript:;" onclick="javascript:submit_form();">Submit</a>');
  var post_send =  ('<label>&nbsp;</label><a id="send-blank">Sending...</a>');

  var validate_form = pre_validate_cf();
  var message = document.getElementById('err-message');
  var form_data = '';
  var form_return = '';

  send_msg.innerHTML = post_send;

  if (validate_form!=0) {

    send_msg.innerHTML = pre_send;
    message.innerHTML = 'Please correct the following errors: <ul>'+validate_form+'</ul>';
    message.style.display = 'block';

  }
  else {

    message.style.display = 'none';

    /* -- SEND TO SERVER -- */
    var name = encodeURIComponent(document.getElementById('li-fname').value);
    var email = encodeURIComponent(document.getElementById('li-email').value);
    var phone = encodeURIComponent(document.getElementById('li-phone').value);
    var company = encodeURIComponent(document.getElementById('li-company').value);
    var address = encodeURIComponent(document.getElementById('li-address').value);
    var promo = encodeURIComponent(document.getElementById('li-sub-promo').value);

    var bus_lighting = is_checked('small-business-lighting');
    var bus_reg_incentives = is_checked('business-registration-incentives');

    form_data = 'name='+name+'&email='+email+'&phone='+phone+'&company='+company+'&address='+address+'&promo_hear='+promo+'&bus_lighting='+bus_lighting+'&bus_reg_incentives='+bus_reg_incentives;

    cf_form_req(form_data);

  }
}

function cf_form_req(params){

  var send_msg = document.getElementById('form-submit');
  var pre_send = ('<label>&nbsp;</label><a href="javascript:;" onclick="javascript:submit_form();">Submit</a>');
  var post_send =  ('<label>&nbsp;</label><a id="send-blank">Sending...</a>');
  var message = document.getElementById('err-message');
  var recaptcha = '';
  recaptcha = grecaptcha.getResponse();
  var params_full = params +'&recaptcha='+ recaptcha;

  send_msg.innerHTML = post_send;

  var url = "/wp-content/themes/li-cdm-lp/lb/api-localbusiness/cf-form-post.php";

  $.ajax({
    url : url,
    type: "POST",
    data : params_full,
    success: function(data)
    {

  console.log(data);

  if(data=='true'){
    document.getElementById('form-validated').style.display = 'block';
    ga('send', 'event', {
      eventCategory: 'Form Submit Success',
      eventAction: 'click',
      eventLabel: event.target.href
    });

    console.log('submit');
  }else {
    message.style.display = 'block';
    send_msg.innerHTML = pre_send;
    message.innerHTML = 'Please correct the following errors: <ul>'+data+'</ul>';
  }

    }
  });
}

function google_captcha_verify(){
  var v = grecaptcha.getResponse();
  if(v.length == 0) { return false; }
  else { return true; }
}

function pre_validate_cf(){

  var name = document.getElementById('li-fname');
  var email = document.getElementById('li-email');
  var phone = document.getElementById('li-phone');
  var company = document.getElementById('li-company');
  var address = document.getElementById('li-address');

  var bus_lighting = is_checked('small-business-lighting');
  var bus_reg_incentives = is_checked('business-registration-incentives');
  var google_captcha = google_captcha_verify();

  //var captcha = document.getElementById('captcha-verify');

  var clean_css = 'border:1px #2ac158 solid';
  var err_css = 'border:1px #FF0000 solid';

  var check = '';
  var log_data = '';

  if(name.value == null || name.value==''){
    check = '<li>Please enter your name.</li>';
    log_data = '[ Empty First Name ],';
    name.style.cssText = err_css;
  }else {
    log_data = '[ First Name: '+name.value+' ],';
    name.style.cssText = clean_css;
  }

  if(email.value == null || email.value==''){
    check += '<li>Please enter your email.</li>';
    log_data += '[ Empty Email ],';
    email.style.cssText = err_css;
  }else if(validate_email(email.value)=='false'){
    check += '<li>Please enter a valid email.</li>';
    log_data += '[ Invalid Email: '+email.value+' ],';
    email.style.cssText = err_css;
  }else {
    log_data += '[ Email: '+email.value+' ],';
    email.style.cssText = clean_css;
  }

  if(phone.value == null || phone.value==''){
    check += '<li>Please enter your phone number.</li>';
    phone.style.cssText = err_css;
  }else if(validate_phone(phone)!=true){
    check += '<li>Please enter a valid phone number. Format: 000-000-0000</li>';
    phone.style.cssText = err_css;
  }else {
    phone.style.cssText = clean_css;
  }

  if(company.value == null || company.value==''){
    check += '<li>Please enter your company name.</li>';
    company.style.cssText = err_css;
  }else {
    company.style.cssText = clean_css;
  }

  if(address.value == null || address.value == ''){
    check += '<li>Please enter your company address</li>';
    address.style.cssText = err_css;
  }else {
    address.style.cssText = clean_css;
  }

  if(google_captcha!== true ){
    check += '<li>Please verify you are not a robot.</li>';
  }

  if(bus_lighting!== 'Checked'){
    if(bus_reg_incentives!=='Checked'){
  check += '<li>Please select a program of interest.</li>';
    }else {

    }
  }

  /* IF VALID -- Clear Check, Set to 0 */
  if(check==''){check=0;}

  return check;
}

function is_checked(id){
  if(document.getElementById(id).checked) {
    return( "Checked" );
  }else {
    return( "Not Checked" );
  }
}

function close_form(){
  document.getElementById('form-validated').style.display = 'none';
}

3.40.3. /wp-content/themes/li-cdm-lp/lb/api-localbusiness/cf-form-post.php

<?php
/* --- CLASS INCLUSION --- */
include('form-clean.php');
$FORMCLEAN = new FORM_CLEAN();

session_start();

$recipients_sbl = array('iamadmin@abc.com');

/* --- PARAMS (Pre-Clean) --- */
$C_NAME                         = $FORMCLEAN->clean($_POST['name']);
$C_EMAIL                        = $FORMCLEAN->clean($_POST['email']);
$C_PHONE                        = $FORMCLEAN->clean($_POST['phone']);
$C_COMPANY                      = $FORMCLEAN->clean($_POST['company']);
$C_ADDRESS                      = $FORMCLEAN->clean($_POST['address']);
$C_PROMO                        = $FORMCLEAN->clean($_POST['promo_hear']);
$C_BUS_LIGHTING = $FORMCLEAN->clean($_POST['bus_lighting']);
$C_BUS_INCENTIVES = $FORMCLEAN->clean($_POST['bus_reg_incentives']);

/* --- VALIDATE --- */
$C_NAME_VALIDATE                                =       $FORMCLEAN->validate($C_NAME,'Name');
$C_EMAIL_VALIDATE                               =       $FORMCLEAN->validate($C_EMAIL,'Email');
$C_PHONE_VALIDATE                               =       $FORMCLEAN->validate($C_PHONE,'Phone');
$C_COMPANY_VALIDATE                             =       $FORMCLEAN->validate($C_COMPANY,'Company');
$C_ADDRESS_VALIDATE                             =       $FORMCLEAN->validate($C_ADDRESS,'Address');


// validate reCaptcha
$C_CAPTCHA_VALIDATE = '';
$C_CAPTCHA = $_POST['recaptcha'];
$C_CAPTCHA_google_request = array(
  'url' => 'https://www.google.com/recaptcha/api/siteverify',
  'options' => array(
    'secret' => 'your_recaptcha_secret_key',
    'response' => $C_CAPTCHA,
  ));

$ch = curl_init();
$options = array(
  CURLOPT_URL => $C_CAPTCHA_google_request['url'],
  CURLOPT_POSTFIELDS => http_build_query($C_CAPTCHA_google_request['options']),
  CURLOPT_FOLLOWLOCATION =>  1,
  CURLOPT_HEADER => 0,
  CURLOPT_RETURNTRANSFER => 1
);
curl_setopt_array($ch, $options);
$C_CAPTCHA_google_result = json_decode(curl_exec( $ch ), true);
curl_close($ch);

if (!(is_array($C_CAPTCHA_google_result) && $C_CAPTCHA_google_result['success'] === true)) {
  $C_CAPTCHA_VALIDATE = ('<li>reCaptcha Match Fails </li>');
}
// validate reCaptcha.

$EMAIL_TO = '';

$SOURCE_CODE = 'Li-Local-Business';

$VALIDATE = str_replace('0','',$C_NAME_VALIDATE.$C_EMAIL_VALIDATE.$C_PHONE_VALIDATE.$C_COMPANY_VALIDATE.$C_ADDRESS_VALIDATE.$C_CAPTCHA_VALIDATE);

if($VALIDATE==''){

  if($C_BUS_LIGHTING=='Checked'){

    /* --- SEND EMAIL (TO ADMINISTRATOR) --- */
    $EMAIL_TO = implode(',', $recipients_sbl);
    //$EMAIL_TO .= 'joseph.alonzi@cleversamurai.com';
    $SUBJECT = '[ Li ] - Small Business Lighting Signup';
    $EMAIL_FROM = 'donotreply@abc.com';

    $MESSAGE = ('
                    <html>
                        <head>
                            <title>[ Li ] - Small Business Lighting Signup</title>
                        </head>
                        <body>

                         <h1>[ Li ] - Small Business Lighting Signup</h1>

                         <b>Name:</b> '.$C_NAME.'<br />
                         <b>Email:</b> '.$C_EMAIL.'<br />
                         <b>Phone:</b> '.$C_PHONE.'<br /><br />

                        <b>How Did You Hear About Us?</b> '.$C_PROMO.'<br /><br />

                         <b>Business Name:</b> '.$C_COMPANY.'<br />
                         <b>Business Address:</b> '.$C_ADDRESS.'<br /><br />

                         The following email: '.$C_EMAIL.' has shown interest in Small Business Lighting.

                        <br />
                        ---
                        <br />
                        Source Code: '.$SOURCE_CODE.$_debug.'

                        </body>
                    </html>
                    ');

    $HEADERS  = 'MIME-Version: 1.0' . "\r\n";
    $HEADERS .= 'Content-type: text/html; charset=iso-8859-1' . "\r\n";
    $HEADERS .= 'From: Li <'.$EMAIL_FROM.'>' . "\r\n";
    mail($EMAIL_TO, $SUBJECT, $MESSAGE, $HEADERS);

  }

  unset($_SESSION['C_CAPTCHA']);
  session_destroy();

  /* --- IF ALL VALIDATES, AND THE EMAIL HAS BEEN SENT --- */
  echo 'true';

}
else {
  echo $VALIDATE;
}

3.40.4. /wp-content/themes/li-cdm-lp/lb/api-localbusiness/form-clean.php

<?php

class FORM_CLEAN{

  public function clean($input){

    /* -- Simple Clean -- */
    $input = preg_replace('#<script(.*?)>(.*?)</script>#is', '', $input);
    $input = str_replace('<?php','',$input);
    $input = str_replace('?>','',$input);
    $input = str_replace("1=1;--",'',$input);
    $input = strip_tags($input);

    return $input;

  }

  public function validate($input,$name){

    $error=0;

    if($input==''){
  $error++;
    }

    if($error!=0){
  return '<li> '.$name.' field is empty or invalid.';
    }else {
  return $error;
    }

  }

  public function check_convert($str){

    if($str=='true'){
  $str='YES';
    }else {
  $str='NO';
    }

    return $str;

  }

}

3.41. TS: Download failed. Destination directory for file streaming does not exist or is not writable

When updating plugins from wp-admin. wp requires the temp folder to be chmod:777 

# create a temp folder with 777 permission outside of the webroot
cd ..
mkdir temp
chmod 777 temp
chown www-data:www-data

Add this line to wp-config.php 

define('WP_TEMP_DIR', ABSPATH . '/../temp/');

3.42. TS: Hacked wp:ts:hacked

https://codex.wordpress.org/FAQ_My_site_was_hacked https://codex.wordpress.org/Hardening_WordPress Change wp users' passwords Compare wp-config.php with wp-config-sample.php wp:file permissions wp:wp-config:DISALLOW_FILE_EDIT

3.42.1. Search backdoor scripts

Use wp:plugin:exploit-scanner to check core and plugin files https://sucuri.net/guides/how-to-clean-hacked-wordpress Used PHP functions base64 str_rot13 gzuncompress eval exec system assert stripslashes preg_replace (with e) move_uploaded_file

3.42.2. Don't execute PHP in wp-content/uploads folder

In wp-content/uploads folder upload this .htaccess 

<Files *.php>
deny from all
</Files>

Find *.php in uploads folder 

find wp-content/uploads/ iname "*.php"

3.43. TS: wp_deregister_script was called incorrectly

Scripts and styles should not be registered or enqueued until the wp_enqueue_scripts, admin_enqueue_scripts, or login_enqueue_scripts hooks. Use something like this 

if ( ! is_admin() ) {
    add_action( 'wp_enqueue_scripts', function(){
        wp_deregister_script( 'jquery' );
        wp_register_script( 'jquery', 'http://ajax.googleapis.com/ajax/libs/jquery/1/jquery.min.js', array(), null, false );
        wp_enqueue_script( 'jquery');
    });
}

3.44. TS: Image editor not working, Edit Media, PHP open tag

Ensure all php files in plugins, mu-plugins, theme's functions.php and its included PHP files:

  • No space before <?php if the file starts with it
  • Files do not end with ?>

3.45. TS: Check list

  • check PHP and MySQL version
  • check WP Core version
  • check proprietary plugins and see which plugins are deprecated. Check PHP and WP version required by plugins.
  • check if there's custom/modified plugins
  • check errors wp:show error
  • check sitemap
  • check website status
  • wp-admin Settings > Reading > Search Engine Visibility > uncheck Discourage search engines from indexing this site.

4. ColdFusion

4.1. cfsetting

Override the server setting for a particular page

requesttimeout
can be overriden in cfquery (database has to support) or cfhttp using the timeout attribute in seconds
<cfsetting enablecfoutputonly="yes|no" 
       requesttimeout="300" 
       showdebugoutput="yes|no">

4.2. Text

FindNoCase("the", "The sentence") returns integer position
0 if not found

4.3. Mail

<cfmail to="1@a.com,2@b.com" from="" subject="" type="text/html">
  <cfmailparam name="X-SMTPAPI" value='{"category": ["abc-xyz"]}' />
  <p>...</p>
</cfmail>

4.4. Function

<cffunction name="getSiteDetail" access="public" returntype="query">
  <cfargument name="intID" type="numeric" default="0">
  <cfset var getSite = "">
  <cfquery name="getSite" datasource="..." dbtype="ODBC">
     SELECT *
     FROM Site
     WHERE ID <> 7
     <cfif argument.intID neq 0>
       AND ID = <cfqueryparam value="#Arguments.intID#" CFSQLType="CF_SQL_NUMERIC">
     </cfif>
     ORDER BY Site.sname
  </cfquery>
  <cfreturn getSite>
</cffunction>

<cfinvoke component="functions.banner" 
          method="getSiteDetail" 
          returnvariable="getSite">
  <cfinvokeargument name="intID" value="#Form.ssite2#">
</cfinvoke>

4.5. Query

  • Attributes
    timeout
    in seconds
<!--- Access --->
<cfquery name="queryname" datasource="abc" dbtype="ODBC">
  SELECT *
  FROM tblOne
</cfquery>

4.5.1. Get Identity After Insert

Access 

<cftransaction>
  <cfquery name="insertQ" datasource="abc">
    INSERT INTO aTable (col1, col2)
    VALUES (1,2)
  </cfquery>
  <cfquery name="getID" datasource="abc">
    SELECT @@identity AS RecordID
  </cfquery>
</cftransaction>

<cfoutput>#getID.recordID#</cfoutput>

TSQL 

<cfquery>
SET NOCOUNT ON;
INSERT INTO aTable (col1,col2)
VALUES (1,2)
SELECT SCOPE_IDENTITY() AS RecordID;
</cfquery>

4.5.2. Insert Null

<cfquery>
  UPDATE tTable SET
  <cfif isDefined('form.formFieldA')>
    intField = <cfqueryparam value="#form.formFieldA#" cfsqltype="cf_sql_numeric">
  <cfelse>
    intField = <cfqueryparam value="" cfsqltype="cf_sql_numeric" null="1">
  </cfif>
</cfquery>

4.6. Fast form to Database

Checkbox 

<input type="checkbox" name="dLocked" value="1">

<cfparam name="form.dLocked" default="0" overwrite="false">
INSERT INTO aTable (dLocked)
VALUES (
<cfqueryparam value="#form.dLocked#" CFSQLType="CF_SQL_BIT">
)

<!--- Insert --->
<form action="site_new.cfm" method="post">
  <input name="Sname" type="text" size="30" maxlength="255">
</form>

// site_new.cfm
<cfinsert datasource="aDB" tablename="TableName">

<!--- Update. cfupdate doesn't work... use cfquery --->
<cfif isDefined("form.phone")> 
    <cfupdate datasource="cfdocexamples" tablename="EMPLOYEES"> 
</cfif> 

<cfquery name="empTable" datasource="cfdocexamples"> 
    SELECT * FROM EMPLOYEES 
</cfquery> 

<!--- This code shows the contents of the employee table and allows you to choose a row for updating. ---> 
<table border="1"> 
<cfoutput query="empTable"> 
    <tr> 
        <td>#firstName#</td> 
        <td>#lastName#</td> 
        <td>#phone#</td> 
        <td><a href="cfupdate.cfm?id=#emp_id#">Edit</a></td> 
    </tr> 
</cfoutput> 
</table> 

<cfif isDefined("url.id")> 
    <cfquery name="phoneQuery" datasource="cfdocexamples"> 
        SELECT * FROM employees WHERE emp_id=#url.id# 
    </cfquery> 
<!--- This code displays the row to edit for update. ---> 
    <cfoutput query="phoneQuery"> 
        <form action="cfupdate.cfm" method="post"> 
        #phoneQuery.firstName# #phoneQuery.lastName#  
        <input name="phone" type="text" value="#phone#" size="12">  
        <input type="submit" value="Update"> 
        <input name="emp_id" type="hidden" value="#emp_id#"> 
        <!--- The emp_id is passed as a hidden field to be used as a primary  
            key in the CFUPDATE. ---> 
        </form> 
    </cfoutput> 
</cfif>

4.7. Loop

<cfloop query="aQueryname" startRow="1" endRow="10" group="Customer_ID">
  #aQueryName.CurrentRow#
</cfloop>

startRow, endRow and group are optional.

Array 

<cfset local.objPath = GetPageContext().getRequest().getServletPath() />
<cfset local.apiNS = ArrayNew(2) />
<cfset local.apiNS[1] = ["v1", "json"] />
<cfset local.apiNS[2] = ["v2", "json2"] />
<cfset local.apiVer = 0 />
<cfset Request.apiCFC = 'json' />

<cfloop array="#local.apiNS#" item="i">
  <cfif FindNoCase('/' & i[1] & '/', local.objPath, 1)>
    <cfset local.apiVer = i[1] />
    <cfset Request.apiCFC = i[2] />
    <cfbreak>
  </cfif>
</cfloop>

4.8. Remote Address

<cfif cgi.REMOTE_ADDR eq "127.0.0.1">

4.9. cftry

<cftry>
  <!--- Do something --->
  <cfcatch type="any">
    <cfdump var="#cfcatch#">
  </cfcatch>
</cftry>

4.10. cfdump

Save cfdump in a string 

<cfsavecontent variable="theDump">
  <cfdump var="#form.uploadSizesAll#" expand="yes" format="text">
</cfsavecontent>
<cfset sError = theDump>

4.11. Javascript

<cfoutput>
  <script>
    var #toscript(form.afield, "jsVarName")#;
  </script>
</cfoutput>

5. Python

5.1. Installation

Version
python -V
(no term)
Windows

5.2. Basics

print, format

Python3 

x = 42

print(f'Hello, {x}')
# eq. to
print('Hello, {}'.format(x))

x = 42
y = 73

if x < y:
print('x < y: x is {} and y is {}'.format(x, y))
(no term)

Condition 

if x > y:
print('x > y')
elif x < y:
print('x < y')
elif x == y:
print('x == y')
else:
print('this is unexpected..')
(no term)

Loop 

words = ['1', '2', '3']

n = 0;
while(n < 5):
print(words[n])
n += 1

a, b = 0, 1

while b < 1000:
print(b, end = ' ', flush = True)
a, b = b, a + b
# a = b
# b = a + b

print() # line ending

for i in words:
print(i)

# from 2 to n-1
for x in range(2, n):
# do something
(no term)

Function 

def abc(n = 1):
print(n)
# return n
x = abc(42)
print(x) # every function return something, if not defined, it returns None
(no term)
Types
  • print(type(x))

  • String 

    x = "abc"
x = 'abc'

x = '''
multi
line
'''
    String methods
    Python3

5.4. Scrapy

  • https://scrapy.org/doc/
  • scrapy runspider a.py -o r.json
  • Create a project
    • scrapy startproject tutorial
    • cd tutorial and put whatevername.py to tutorial/spiders
    • scrapy crawl quotes
    • scrapy crawl quotes -o r.json
    • scrapy crawl quotes -o r.csv -t csv
  • Modify tutorial/settings.py
    • ROBOTSTXT_OBEY = False
    • USER_AGENT = 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
    • AUTOTHROTTLE_ENABLED = True

6. CSS

6.1. Display, Float, Position, box-sizing

6.1.1. Display

  • block
  • inline
    • width height have no effect
    • margin only can be set horizontally
  • inline-block
    • width height can be set
    • spaces between inline-block's can be eliminated by font-size:0;

6.1.2. Float

  • Clear float to fix parent height less than total children height 

    <div class="clearfix">
  <div style="">Div 1 doesn't float</div>
  <div style="float: left;">Div 2</div>
</div>

    /* Solution 1 */
.clearfix:after {
content: " ";
display:block;
clear:both;
height:0;
}

/* Solution 2 */
.clearfix {
display:inline-block;
width:100%;
}

/* Solution 2 */
.clearfix {
overflow:hidden;
}

6.1.3. Position

static
Default position. left, right, top, bottom have no effect
relative
is relative to default static position, use left, right, top and bottom
  • Other elements adjacent to position:relative element will not be adjusted to fit into any gap left by the element (treat the relative element as static)
fixed
position:fixed is relative to viewport
  • Its element is removed from the normal document flow, and no space is created for the element in the page layout
  • It's positioned relative to the initial containing block established by the viewport, except when one of its ancestors has a transform, perspective, or filter property set to something other than none, in which case that ancestor behaves as the containing block. Its final position is determined by the values of top, right, bottom, left
  • In printed documents, the element is placed in the same position on every page
(no term)
sticky
  • It is positioned according to the normal flow of the document, and then offset relative its nearest scrolling ancestor and containing block (nearest block-level ancestor) including table-related elements
  • scrolling ancestors are overflow is hidden, scroll, auto or overlay
  • It is positioned based on the value of top, right, bottom and left
absolute
postion:absolute is positioned relative to the nearest positioned ancestor. If there's no positioned ancestors, document body is used
  • The showing area of position:absolute elements depend on the outer most parent's width/height and padding which is <body>
positioned element
its position is anything except static
  • When elements are positioned, they can overlap other elements
  • it creates a stacking context
(no term)
Stacking context
(no term)
CSS Position from MDN

6.1.4. box-sizing

box-sizing: border-box | padding-box | content-box (default);

6.2. Horizontal Center Absolute

The absolute element is positioned related to the first position:relative parent

Center that abosulte element in relate to that parent

Notice: any parents will not expand in width and height according to that absolute child element 

.grandchild {
  position: absolute;
  margin-left: auto;
  margin-right: auto;
  left: 0;
  right: 0;
}

6.3. Transform, Transition

6.3.1. Transition

  • transition: /transition-property/ /transition-duration/ /transition-timing-function/ /transition-delay/ 

    div {
  width: 100px;
  height: 100px;
  background: red;
  transition: width 2s linear 3s, height 2s linear 3s, transform 2s linear 3s;
}
    transition-property
    can be 'all'
    transition-timing-function
    default linear
    ease-in
    slow start
    ease-out
    slow end
    (no term)
    e.g. use ease-in for div:hover and ease-out in div
    transition-delay
    delay n seconds before the current transition starts.

6.3.2. Transform 

div:hover {
  width: 300px;
  height: 300px;
  transform: 
    rotate(180deg) 
    translate(-20px, 0)
    scale(0.9, 2)
    skew(30deg, 20deg);
}
/* You can turn off all transform */
/* transform: none; */

6.4. Insert CSS File css:js insert

['//ajax.googleapis.com/ajax/libs/jqueryui/1.10.2/themes/ui-darkness/jquery-ui.css',
 '/wp-content/themes/xxx/css/interstitial.css'].forEach(function (src) {
     var _css = document.createElement("link");
     _css.rel = 'stylesheet';
     _css.type = 'text/css';
     _css.href = src;
     document.head.appendChild(_css);
 });

6.5. Font, Text css:font

6.5.1. Ellipsis

Single Line 

div.test {
 white-space: nowrap;
 width: 200px; // width has to be defined
 overflow: hidden; // needed
 text-overflow: ellipsis; // change to inherit when hover then the clipped text will show
}

Multiple Lines (IE and Edge will not show 3 dots) 

.excerpt {
  display: block;
  display: -webkit-box;
  font-weight: normal;
  font-size: 15px;
  line-height: 1.4;
  -webkit-line-clamp: 9;
  -webkit-box-orient: vertical;
  overflow: hidden;
  text-overflow: ellipsis;
  height: 189px; /* 15*1.4*9 */
}

6.5.2. Long Word

word-wrap: break-word; // allow to break a long word

6.5.3. Font size, line-height, letter-spacing

  • font-size in child elements inherit parent element
  • em uses the current element's font-size or its closest parent element which has font-size defined
  • rem is relative to the <html>'s font-size
    • e.g. set rem for line-height in <body> to ensure line-height is relative to <html>'s font-size
  • Both em and rem will be affected by parent element's css:zoom in Edge!
  • vw and vh are the viewport width and height. 1vw is 1% of vw
  • Google recommends line-height is at least 1.2
    • line-height is normal means to use user agent style. Most likely default is 1 not 1.2
  • letter-spacing, padding using em is relative to the current element's font size

6.5.4. Prevent sup and sub affecting line height css:sup:line-height

sup, sub {
  vertical-align: baseline;
  position: relative;
  top: -0.4em;
}
sub { 
  top: 0.4em; 
}

Email 

.sup {
  line-height:0;
  font-size:6px;
  vertical-align:5px;
}

6.5.5. Fluid Typography

body {
  font-size: calc([minimum size] + ([maximum size] - [minimum size]) * ((100vw - [minimum viewport width]) / ([maximum viewport width] - [minimum viewport width])));
}

// example
html {
  font-size: 16px; /* [minimum size] */
}

/* 320px = [minimum viewport width] */
@media screen and (min-width: 320px) { 
  html {
    font-size: calc(16px + (22 - 16) * ((100vw - 320px) / (1000 - 320) ) );
  }
}

/* 1000px = [maximum viewport width] */
@media screen and (min-width: 1000px) {
  html {
    font-size: 22px; /* [maximum size] */
  }
}

https://css-tricks.com/fitting-text-to-a-container/

6.5.6. @font-face css:font-face

  • Most widely accepted font format is WOFF2, WOFF then TTF and OTF
    • On macOS
      • the Font Book.app can only add TTF, TTC, and OTF
      • If you find macOS renders thicker glyphs for light text on dark background thicker than other browsers, change:
        • -webkit-font-smoothing to be antialiased for Safari and Chrome
        • -moz-osx-font-smoothing to be grayscale for Firefox
        • Do nothing for Windows and Linux
      • -webkit-font-smoothing
        auto
        Uses subpixel anti-aliasing when available; this is the default
        none
        Turn font smoothing off; display text with jagged sharp edges
        antialiased
        Smooth the font on the level of the pixel, as opposed to the subpixel. Switching from subpixel rendering to anti-aliasing for light text on dark backgrounds makes it look lighter.
        • eq. to Firefox's grayscale -moz-osx-font-smoothing: grayscale;
        subpixel-antialiased
        On most non-retina displays, this will give the sharpest text.
      • -moz-osx-font-smoothing
        auto
        Allow the browser to select an optimization for font smoothing, typically grayscale.
        grayscale
        Render text with grayscale anti-aliasing, as opposed to the subpixel. Switching from subpixel rendering to anti-aliasing for light text on dark backgrounds makes it look lighter.
      • *-font-smoothing has no effect, so the default antialising mode is subpixel-antialiasing
      • In short, you need a different font source file for macOS
  • IE 11 and less only supports EOT. Edge supports WOFF
  • Use full url in src: url() format() for email HTML
    format()
    Optional. Can be: woff2, woff, truetype, opentype, embedded-opentype, svg
  • For a Variable Font, can be a range e.g.
    • font-weight: 100 900;
    • font-stretch: 25% 140%;
@font-face {
    font-family: myFirstFont;
    src: url(sansation_light.woff) format('woff');
}

/* Another for bold */
@font-face {
    font-family: myFirstFont;
    src: url(sansation_bold.woff) format('woff');
    font-weight: bold;
/*
 font-stretch: normal | condensed | semi condensed | extra condensed | ultra condensed | expanded | ...;
 font-style: normal | italic | oblique 
*/
}
6.5.6.1. font-display under font-face
  • Chrome and Firefox will use the font fallback if the first font choice is not available after 3 seconds (timeout). If the first font choice is available later, a swap occurs
  • IE has 0 timeout and use fallback immediately and swap later if font is available
  • Safari always wait for the font to be downloaded
  • Not supported in IE, Edge and iOS < 11.4
    auto
    use user-agent's default. Most user-agents use block
    block
    After 3 seconds if the font is not loaded use a invisible text and swap to the downloaded font with infinite swap period
    swap
    browser draws text immediately with a fallback and swap to it after it's loaded.
    fallback
    fallback at first and swap to it later. If too much time passes (3s), then the fallback is used for the rest of page lifetime.
    optional
    the font is a nice to have but not critical. It leaves it up to the browser to decide whether to initiate the font download
@font-face {
  font-family: 'Awesome Font';
  font-style: normal;
  font-weight: 400;
  font-display: auto; /* or block, swap, fallback, optional */
  src: local('Awesome Font'),
       url('/fonts/awesome-l.woff2') format('woff2'), /* will be preloaded */ 
       url('/fonts/awesome-l.woff') format('woff'),
       url('/fonts/awesome-l.ttf') format('truetype'),
       url('/fonts/awesome-l.eot') format('embedded-opentype');
  unicode-range: U+000-5FF; /* Latin glyphs */
}
6.5.6.2. Font Loading API
var font = new FontFace("Awesome Font", "url(/fonts/awesome.woff2)", {
  style: 'normal', unicodeRange: 'U+000-5FF', weight: '400'
});

// don't wait for the render tree, initiate an immediate fetch!
font.load().then(function() {
  // apply the font (which may re-render text and cause a page reflow)
  // after the font has finished downloading
  document.fonts.add(font);
  document.body.style.fontFamily = "Awesome Font, serif";

  // OR... by default the content is hidden, 
  // and it's rendered after the font is available
  var content = document.getElementById("content");
  content.style.visibility = "visible";

  // OR... apply your own render strategy here... 
});

// check status :: unloaded, loading, loaded, error
font.status

// Promise loaded resolves when the font is loaded
notoSansRegular.loaded.then((fontFace) => {
  // This is where you can declare a new font-family, because the font is now loaded and ready.  
  console.info('Current status', fontFace.status);
  console.log(fontFace.family, 'loaded successfully.');
  // Throw an error if loading wasn't successful
}, (fontFace) => {
  console.error('Current status', notoSansRegular.status);
});


// when all fonts are ready
document.fonts.ready.then((fontFaceSet) => {
  console.log(document.fonts.size, 'FontFaces loaded.');
});
6.5.6.3. Bootstrap 3
@font-face {
  font-family: 'Glyphicons Halflings';
  src: url('../fonts/glyphicons-halflings-regular.eot');
  src: url('../fonts/glyphicons-halflings-regular.eot?#iefix') format('embedded-opentype'), 
       url('../fonts/glyphicons-halflings-regular.woff2') format('woff2'), 
       url('../fonts/glyphicons-halflings-regular.woff') format('woff'), 
       url('../fonts/glyphicons-halflings-regular.ttf') format('truetype'),
       url('../fonts/glyphicons-halflings-regular.svg#glyphicons_halflingsregular') format('svg');
}
6.5.6.4. Google Font and local() unicode-range
  1. local() unicode-range
    • Example <link href="https://fonts.googleapis.com/css?family=Montserrat:400,700" rel="stylesheet" type="text/css">
    • if user agent has the font, then the font will not be downloaded css:font-face:local
    • ranges are separated by comma
      Sigle codepoint
      U+416
      Interval range
      start and end e.g. U+400-4ff
      Wildcard range
      ? is any hexadecimal digit e.g. U+4??
    • benefit of loading variants of the same font
      • Can't make a bold font lighter
      • Can't make an oblique (italic) font "straight"
      • Has limited ability to synthesize bold(er) fonts
      • Has limited ability to synthesize oblique fonts and may produce wrong shapes esepcially in Cyrillic fonts
    /* cyrillic-ext */
@font-face {
  font-family: 'Montserrat';
  font-style: normal;
  font-weight: 400;
  src: local('Montserrat Regular'), local('Montserrat-Regular'), url(https://fonts.gstatic.com/s/montserrat/v12/JTUSjIg1_i6t8kCHKm459WRhyzbi.woff2) format('woff2');
  unicode-range: U+0460-052F, U+1C80-1C88, U+20B4, U+2DE0-2DFF, U+A640-A69F, U+FE2E-FE2F;
}
/* cyrillic */
@font-face {
  font-family: 'Montserrat';
  font-style: normal;
  font-weight: 400;
  src: local('Montserrat Regular'), local('Montserrat-Regular'), url(https://fonts.gstatic.com/s/montserrat/v12/JTUSjIg1_i6t8kCHKm459W1hyzbi.woff2) format('woff2');
  unicode-range: U+0400-045F, U+0490-0491, U+04B0-04B1, U+2116;
}
/* vietnamese */
@font-face {
  font-family: 'Montserrat';
  font-style: normal;
  font-weight: 400;
  src: local('Montserrat Regular'), local('Montserrat-Regular'), url(https://fonts.gstatic.com/s/montserrat/v12/JTUSjIg1_i6t8kCHKm459WZhyzbi.woff2) format('woff2');
  unicode-range: U+0102-0103, U+0110-0111, U+1EA0-1EF9, U+20AB;
}
/* latin-ext */
@font-face {
  font-family: 'Montserrat';
  font-style: normal;
  font-weight: 400;
  src: local('Montserrat Regular'), local('Montserrat-Regular'), url(https://fonts.gstatic.com/s/montserrat/v12/JTUSjIg1_i6t8kCHKm459Wdhyzbi.woff2) format('woff2');
  unicode-range: U+0100-024F, U+0259, U+1E00-1EFF, U+2020, U+20A0-20AB, U+20AD-20CF, U+2113, U+2C60-2C7F, U+A720-A7FF;
}
/* latin */
@font-face {
  font-family: 'Montserrat';
  font-style: normal;
  font-weight: 400;
  src: local('Montserrat Regular'), local('Montserrat-Regular'), url(https://fonts.gstatic.com/s/montserrat/v12/JTUSjIg1_i6t8kCHKm459Wlhyw.woff2) format('woff2');
  unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD;
}
/* cyrillic-ext */
@font-face {
  font-family: 'Montserrat';
  font-style: normal;
  font-weight: 700;
  src: local('Montserrat Bold'), local('Montserrat-Bold'), url(https://fonts.gstatic.com/s/montserrat/v12/JTURjIg1_i6t8kCHKm45_dJE3gTD_u50.woff2) format('woff2');
  unicode-range: U+0460-052F, U+1C80-1C88, U+20B4, U+2DE0-2DFF, U+A640-A69F, U+FE2E-FE2F;
}
/* cyrillic */
@font-face {
  font-family: 'Montserrat';
  font-style: normal;
  font-weight: 700;
  src: local('Montserrat Bold'), local('Montserrat-Bold'), url(https://fonts.gstatic.com/s/montserrat/v12/JTURjIg1_i6t8kCHKm45_dJE3g3D_u50.woff2) format('woff2');
  unicode-range: U+0400-045F, U+0490-0491, U+04B0-04B1, U+2116;
}
/* vietnamese */
@font-face {
  font-family: 'Montserrat';
  font-style: normal;
  font-weight: 700;
  src: local('Montserrat Bold'), local('Montserrat-Bold'), url(https://fonts.gstatic.com/s/montserrat/v12/JTURjIg1_i6t8kCHKm45_dJE3gbD_u50.woff2) format('woff2');
  unicode-range: U+0102-0103, U+0110-0111, U+1EA0-1EF9, U+20AB;
}
/* latin-ext */
@font-face {
  font-family: 'Montserrat';
  font-style: normal;
  font-weight: 700;
  src: local('Montserrat Bold'), local('Montserrat-Bold'), url(https://fonts.gstatic.com/s/montserrat/v12/JTURjIg1_i6t8kCHKm45_dJE3gfD_u50.woff2) format('woff2');
  unicode-range: U+0100-024F, U+0259, U+1E00-1EFF, U+2020, U+20A0-20AB, U+20AD-20CF, U+2113, U+2C60-2C7F, U+A720-A7FF;
}
/* latin */
@font-face {
  font-family: 'Montserrat';
  font-style: normal;
  font-weight: 700;
  src: local('Montserrat Bold'), local('Montserrat-Bold'), url(https://fonts.gstatic.com/s/montserrat/v12/JTURjIg1_i6t8kCHKm45_dJE3gnD_g.woff2) format('woff2');
  unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD;
}
  2. Google Fonts API - CSS
  3. Google Web Font Loader - JavaScript
    • https://github.com/typekit/webfontloader
      • A JavaScript library to dynamic font loading Google and Typekit fonts

    • Synchronous load 

      <script src="https://ajax.googleapis.com/ajax/libs/webfont/1.6.26/webfont.js"></script>
<script>
  WebFont.load({
google: {
  families: ['Droid Sans', 'Droid Serif', 'Roboto:300,400,700', ]
}
  });
</script>

    • Async load 

      WebFontConfig = {
google: {
    families: ['Poppins:400,500']
}
};
(function(d) {
if ( typeof WebFont === 'undefined' ) {
    var wf = d.createElement('script'), s = d.scripts[0];
    wf.src = 'https://ajax.googleapis.com/ajax/libs/webfont/1.6.26/webfont.js';
    wf.async = true;
    s.parentNode.insertBefore(wf, s);
} else {
    WebFont.load(WebFontConfig);
}
})(document);
6.5.6.5. Web Safe Fonts css:font-face:web safe fonts
  • Arial, Arial Black
  • Comic Sans MS
  • Courier New
  • Georgia
  • Impact / Charcoal
  • Lucida Sans / Lucida Grande
  • Tahoma / Geneva
  • Times New Roman
  • Trebuchet MS
  • Verdana

Web Safe Fonts in PC and Mac and font stack

6.5.7. font-feature-setting css:font-feature-settings

  • Some fonts have OpenType Features. Refer to font:woff
/* Use the default settings */
font-feature-settings: normal;

/* Set values for OpenType feature tags */
font-feature-settings: "smcp"; /* same as font-feature-settings: "smcp" 1; */
font-feature-settings: "smcp" on; /* same as font-feature-settings: "smcp" 1; */

font-feature-settings: "swsh" 2;
font-feature-settings: "smcp", "swsh" 2;

/* Global values */
font-feature-settings: inherit;
font-feature-settings: initial;
font-feature-settings: unset;

/* Syntax
 normal | <string> [ <integer> | on | off ]?
*/

6.5.8. font-variation-settings css:font-variation-settings

  • Change value for each axis for a Variable Font 

    font-variation-settings: "wght" 370;
font-variation-settings: "wght" 370, "wdth" 75, "YOPQ" 54;

  • Fallback 

    h1 {
font-family: some-non-variable-font-family;
}
@supports (font-variation-settings: 'wdth' 115) {
h1 {
    font-family: some-variable-font-family;
}
}

6.6. CSS Selectors CSS Selectors

div > p
select p elements where the direct parent is div
div + p

select p elements that are placed immediately after div 

<div></div>
<p></p>
p ~ ul

select ul elements that are preceded by a p element 

<p></p>
<ul></ul>
<ul></ul>
a[target]
<a>'s with target attribute
  • a[target=_blank]
a[title~=flower]
containing a word or followed by a hypen (not match flowers)
a[lang|=en]
starting with a word or followed by a hyphen (en, en-us)
div[class^="test"]
starting with (anything)
div[class$="test"]
ending with (anything)
a[href*="w3schools"]
containing
div[id="c1"], div[id="c2"]
id should be c1 or c2
div[id^="c1"][lang^="en-"]
id starts with c1 AND lang starts with en-
(no term)

:root 

:root {
  /* matches <html> */
  /* perfect place to define custom properties to be used by var() function */
  --example-name: 5px 24px;
  --blue: #007bff;
}
(no term)
a:active
(no term)
a:hover
a:link
Unvisited and normal. div:link may be possible when div has href attribute
(no term)
a:visited
(no term)
p:even p:odd ::*jQuery only*
(no term)
p:first p:last
p::after
<img> can't have pseudo element
(no term)
p::before
(no term)
p::first-letter
(no term)
p::first-line
(no term)
p::selection
p::lang(it)
language attribute
div::selection
the selected portion of an element (e.g. select text)
(no term)
input[type=text]::placeholder
(no term)
input:checked
(no term)
input:enabled, input[type="text"]:disabled
input[type=radio]:default, input[type=checkbox]:default
default checked radio or checkbox
(no term)
input:focus
(no term)
input:invalid, input:valid
input:optional
without required attribute
input:required
with required attribute
p:empty
has no children including text nodes
div:not(p)
all children elements except <p>'s
div:not(.aClass)
all div elements except the ones with class aClass
p:eq(2)
jQuery only index at 2, starting from 0
p:gt(2) p:lt(2)
jQuery only index greater than 2
p:contains('abc')
jQuery only contains abc text
p:parent
jQuery only Select p elements which are parents (has some children or text node)
div:has(p[class=a])
jQuery only select div which has p.a
p:first-child
<p>'s are the first child of any elements
(no term)
p:last-child
p:nth-child(2)
start with 1
p:nth-child(7n-1)
n starts with 1
(no term)
tr:nth-child(even)
(no term)
tr:nth-child(odd)
(no term)
p:nth-last-child(2)
p:only-child
<p>'s are the only child of any elements
div p:first-of-type
<div> has 4 children <p>'s and first child is <a>. Select 1st of <p> children.
(no term)
div p:last-of-type
div p:nth-of-type(2)
can use n like nth-child
(no term)
div p:only-of-type
#news:target
anchor name id=news matches the URL hash
(no term)
ul > li:not(:last-child):after

6.7. Specificity

(a, b, c, d) start with (0, 0 , 0 ,0) a - inline add 1 b - each id add 1 c - class, pseudo-class and attribute add 1 d - element add 1

Pseudo-class :not is not considered as pesudo-class in specificity calculation. But anything inside it counts.

6.8. Functions

6.8.1. attr(data-attr-1) css:f:attr

a::after {
 content: " (" attr(href) ")";
}

6.8.2. calc() css:f:calc

  • No parenthesis is allowed inside calc()
  • Only css:f:attr and calc() are allowed inside calc()
/* Always leave 50px on both sides for space
 and take all that is left for width
*/
div {
 width: calc(100% - 100px); /* calculate */
 margin-left: 50px;
}

/* 3 columns minus gutter */
.item {
  width: calc(100%/3 - 10px/3);
}

6.8.3. counter css:f:counter

body {
 counter-reset: section;
}
h1 {
 counter-reset: subsection;
}
h1::before {
 counter-increment: section,
 content: "Section " counter(section) ". ";
}
h2::before {
 counter-increment: subsection;
 content: counter(section) "." counter(subsection) " ";
}

6.8.4. var(x) css:f:var

:root {
    --body-font-family: Arial;
    --amstel-wght: 370;
    --amstel-wdth: 75;
}

* {
    font-variable-settings: 'wght' var(--amstel-wght), 'wdth' var(--amstel-wdth);
}

body {
    font-family: var(--body-font-family);
}

6.8.5. minmax(x,y) css:f:minmax

/*
sets minimum row height to 100px, max height to auto
*/
grid-column-rows: minmax(100px, auto);

6.8.6. repeat(x,y) css:f:repeat

grid-template-columns: repeat(3, 20px [col-start]) 5%;

6.9. border-image

border-image: source slice width outset repeat|initial|inherit; border-image: none 100% 1 0 stretch

6.10. background css:background

background: bg-color bg-image position/bg-size bg-repeat bg-origin bg-clip bg-attachment initial|inherit;

background: url(img_flwr.gif) right bottom no-repeat, url(paper.gif) left top repeat;

6.10.1. background-position

background-position: right center; /* align right on X and align center on Y */
background-position: right bottom;
background-position: right 100px bottom -100px; /* for right/left or top/bottom, +- pixels */
background-position: calc(50% - 100px) calc(50% + 100px); /* for center, calc has be used */

6.10.2. background-size

background-size: 20px auto;
background-size: 50% auto; /* sets the width of the bg image in percent of the parent element. */
background-size: contain; /* bg image might not cover all content area */
background-size: cover; /* content area might not see the whole bg image */

/* Stretch on y-axis and repeat on x-axis */
background: url() center repeat-x;
background-size: auto 100%;

6.10.3. background-clip, background-origin

background-clip: border-box; / default. largest area / padding-box, content-box

/ For background image background-origin: border-box; / default. largest area // padding-box, content-box

6.10.4. background-attachment

default scroll
bg is fixed with regard to the element itself and does not scroll with its contents (bg is attached to the element's border). Fixed to document window.
fixed
bg is fixed with regard to the viewport. Even if an element has a scrolling mechanism. Bg doesn't move with the element
local
bg is fixed with regard to the element's contents: if the element has a scrolling mechanism, the bg scrolls with the element's contents. Fixed to both viewport and document window.

6.10.5. Add dark layer on a background image

15% darker 

/* Before */
background: url(/slide-03.jpg) no-repeat center center;

/* After */
background: linear-gradient(to bottom, rgba(0, 0, 0, 0.15) 0%, rgba(0, 0, 0, 0.15) 100%), url('/slide-03.jpg') no-repeat center center;

6.10.6. Mobile background-file:///Users/lili/li/notes/fixed background-size:cover

Instead of 

.cs-hero-background {
  background: url('...') no-repeat center bottom fixed;
  -webkit-background-size: cover;
  -moz-background-size: cover;
  -o-background-size: cover;
  background-size: cover;
}

Do this, body:before might work in some situation 

.cs-hero-background:before {
  content: "";
  display: block;
  position: fixed;
  left: 0;
  top: 0;
  width: 100%;
  height: 100%;
  z-index: -10;
  background: url('...') no-repeat center bottom;
  -webkit-background-size: cover;
  -moz-background-size: cover;
  -o-background-size: cover;
  background-size: cover;
}

6.11. List style

<'list-style-type'> || <'list-style-position'> || <'list-style-image'> Can't control the image size.

6.12. Color

6.12.1. CSS inherit color

You can grab the parent `color` css property and use it in any color properties in child element 

<div class="parent">
  <div class="child">
  </div>
</div>

<style>
  .parent {
  color: red;
  background-color: blue;
  }
  .child {
  background-color: currentColor; /* take the parent color: red */
  color: currentColor; /* Take the parent color: red */
  }  
</style>

6.12.2. rgb(), rgba() CSS rgba

#f03
#ff0033
#FF0033
rgb(255, 0, 51) // no fraction all integers
rgb(100%, 0%, 20%) // all % no integer

#f030     // 0% opaque red
#ff003300 // 0% opaque red
#FF003388 // 50% opaque red
rgba(255,0,0,0.7) // 70% opaque red

6.12.3. hsl(), hsla()

Hue :: integer, angle degree of the color circle. Red is 0, Green is 120, Blue is 240 Saturation :: percentage. 0% is grey Lightness :: 100% is white, 0% is black, 50% is normal 

hsla(240, 100%, 50%, 0.05) // 5% opaque blue

6.12.4. Darker or lighter color

http://www.0to255.com/cd1713

6.13. Gradient and repeat gradient

Both CSS gradient and CSS repeat gradient can be stacked in background-image 

linear-gradient(45deg, blue, red);

/* to bottom, to top
   to right, to left
   to bottom right, to top left
*/
/* blue at 0%, white at 80%, orange at 100% */
linear-gradient(to bottom, blue, white 80%, orange);

/* evenly distributed */
linear-gradient(to right, red, orange, yellow, green, blue);

/* Use it with background image */
background: linear-gradient(to right, rgba(255,255,255,0), rgba(255,255,255,1))
       ,url(/bg.jpg);

/* non evenly distributed :: 0-50% solid black, 50%-100% solid transparent */
background: linear-gradient(to right, black 0%, black 50%, transparent 50%, transparent 100%);

/* evenly spaced */
radial-gradient(red, yellow, rgb(30, 144, 255));

/* non evenly distributed */
radial-gradient(red 5%, yellow 25%, #1E90FF 50%);

/* With shape :: circle or ellipse
   Default farthest-sided
   Fade from the center point to the ___
   closest-corner, closest-side, farthest-corner, farthest-side
*/
radial-gradient(circle, yellow, #f06d06);

/* gradient from center point to corner */
radial-gradient(
 circle farthest-corner at top right,
 yellow, #f06d06
);

Repeat gradient 

/ On top of percentage, px can be used in repeat gradient */
repeating-linear-gradient(-45deg, red, red 5px, white 5px, white 10px);

6.14. opacity

0, 0.3, 1. The lower the value the more transparent Child elements inherits the parent opacity value If you don't want to carry opacity to children, define rgba color in parent elements CSS rgba

6.15. Shadow

Shadows can be stacked 

box-shadow: h-shadow v-shadow blur spread color |inset|initial|inherit;
text-shadow: h-shadow v-shadow blur-radius color|none|initial|inherit;

Text stroke/outline 

text-shadow:
  -1px -1px 0 #000,
   0   -1px 0 #000,
   1px -1px 0 #000,
   1px  0   0 #000,
   1px  1px 0 #000,
   0    1px 0 #000,
  -1px  1px 0 #000,
  -1px  0   0 #000;

// you might need to adjust font smoothing to antialiased
font-smoothing: antialiased;
-webkit-font-smoothing: antialiased;

// there's a new way! Won't work on IE but work on all current browsers
-webkit-text-fill-color:#ffd33a;
-webkit-text-stroke: 1px #ffd33a;

6.16. object-fit

Apply this to <img>, <video> to have background effect. Usually width and height are set in those elements.

fill (default) | contain | cover | none | scale-down 

<img src="..." class="cover">

img {
  width: 150px;
  height: 100px;
}
.cover {
  -webkit-object-fit: cover;
  -moz-object-fit: cover;
  -ms-object-fit: cover;
  -o-object-fit: cover;
  object-fit: cover;
}

Edge and IE only works for <img> element

6.17. CSS target IE 11 and IE10

Target IE 11 only 

_:-ms-fullscreen, :root .msie11 { color: blue; }

Target IE 10 and IE 11 

@media all and (-ms-high-contrast:none) {
  .foo { color: green } /* IE10 */
  *::-ms-backdrop, .foo { color: red } /* IE11 */
}

6.18. @media css:@media

6.18.1. Syntax

  • @media not|only /mediatype/ and (/media feature/) {}
  • html:meta:viewport
  • <link rel="stylesheet" media="mediatype and|not|only (media feature)" href="style.css">
  • css:@media:type
    Possible types
    all, print, screen, speech
    (no term)
    not /mediatype/ only /mediatype/
    (no term)

    They can be nested with OR (separated by comma) but not and 

    /* This works */
@media screen and (min-width:200px), not print and (min-width:300px) {}

/* This syntax is wrong */
@media screen and (min-width:200px) and not print and (min-width:300px) {}
    (no term)
    Nested media queries (using commas) is supported on all browsers except IE11-
    Note
    not /mediatype/ is different from not (/mediafeature/)
  • Media feature
    • Each media feature, key:value or key (Some media feature doesn't have a value)
    • must be wrapped in ()

6.18.2. Testing

var mql = window.matchMedia("(orientation: portrait)");
// "(max-width: 600px)"
// window.addListener(handlerOrientationChange);
handlerOrientationChange(mql);

function handlerOrientationChange(mql) {
 if (mql.mathces) {
  console.log('media query matches');
 }
 else {
  console.log('media query does not match');
 }
}

6.18.3. Features css:@media:features

min-width, max-width and:

Key Min/Max? value
orientation no landscape, portrait. Not reliable when soft keyboard opens
color yes int, number of colors. (color) for color device
aspect-ratio yes int1/int2
grid no 0, 1 or (grid). The last suggests it's a device with 1 font
width, height yes viewport
monochrome yes 0 (not mono) or int
resolution yes 300dpi or 2dppx
pointer no fine (mouse, drawying stylus), coarse (finger), none (not a pointing device)
hover no none (finger), hover (can have hover state)
  • pointer and hover always refer to the primary device while any-pointer and any-hover refer to any if exists
  • any-hover has on-demand which means mobile services can emulate hovering when the user performs a long tap
  • Don't use min/max-device-width as it refers to the screen size not the viewport size

6.18.4. Responsive Design

6.18.4.1. Mostly Fluid

https://googlesamples.github.io/web-fundamentals/fundamentals/design-and-ux/responsive/mostly-fluid.html 

<div class="container" role="main">
  <div class="c1">
  </div>
  <div class="c2">
  </div>
  <div class="c3">
  </div>
  <div class="c4">
  </div>
  <div class="c5">
  </div>
</div>

.container {
  display: -webkit-flex;
  display: flex;
  -webkit-flex-flow: row wrap;
  flex-flow: row wrap;
}

.c1, .c2, .c3, .c4, .c5 {
  width: 100%;
}

@media (min-width: 600px) {
  .c2, .c3, .c4, .c5 {
    width: 50%;
  }
}

@media (min-width: 800px) {
  .c1 {
    width: 60%;
  }
  .c2 {
    width: 40%;
  }
  .c3, .c4, .c5 {
    width: 33.33%;
  }
}

@media (min-width: 800px) {
  .container {
    width: 800px;
    margin-left: auto;
    margin-right: auto;
  }
}
6.18.4.2. Column Drop

https://googlesamples.github.io/web-fundamentals/fundamentals/design-and-ux/responsive/column-drop.html 

<div class="container" role="main">
  <div class="c1"></div>
  <div class="c2"></div>
  <div class="c3"></div>
</div>

.container {
  display: -webkit-flex;
  display: flex;
  -webkit-flex-flow: row wrap;
  flex-flow: row wrap;
}

.c1, .c2, .c3 {
  width: 100%;
}

@media (min-width: 600px) {
  .c1 {
    width: 60%;
    -webkit-order: 2;
    order: 2;
  }

  .c2 {
    width: 40%;
    -webkit-order: 1;
    order: 1;
  }

  .c3 {
    width: 100%;
    -webkit-order: 3;
    order: 3;
  }
}


@media (min-width: 800px) {
  .c2 {
    width: 20%;
  }

  .c3 {
    width: 20%;
  }
}
6.18.4.3. Layout shifter

https://googlesamples.github.io/web-fundamentals/fundamentals/design-and-ux/responsive/layout-shifter.html 

<div class="container" role="main">
  <div class="c1"></div>
  <div class="c4">
    <div class="c2"></div>
    <div class="c3"></div>
  </div>
</div>

.container {
  display: -webkit-flex;
  display: flex;
  -webkit-flex-flow: row wrap;
  flex-flow: row wrap;
}

.c1, .c2, .c3, .c4 {
  width: 100%;
}

@media (min-width: 600px) {
  .c1 {
    width: 25%;
  }

  .c4 {
    width: 75%;
  }

}

@media (min-width: 800px) {
  .container {
    width: 800px;
    margin-left: auto;
    margin-right: auto;
  }
}
6.18.4.4. Off canvas

https://googlesamples.github.io/web-fundamentals/fundamentals/design-and-ux/responsive/off-canvas.html 

<div class="container" role="main">
  <div class="c1" id="leftDrawer">
  </div>
  <div class="c2" id="mainPanel">
    Click the <code>div</code>'s to change views
  </div>
  <div class="c3" id="rightDrawer">
  </div>
</div>

body {
  overflow-x: hidden;
}

.container {
  display: block;
}

.c1, .c3 {
  position: absolute;
  width: 250px;
  height: 100%;

  /*
    This is a trick to improve performance on newer versions of Chrome
    #perfmatters
  */
  -webkit-backface-visibility: hidden;
  backface-visibility: hidden; 

  -webkit-transition: -webkit-transform 0.4s ease-out;
  transition: transform 0.4s ease-out;

  z-index: 1;
}

.c1 {
  /*
  Using translate3d as a trick to improve performance on older versions of Chrome
  See: http://aerotwist.com/blog/on-translate3d-and-layer-creation-hacks/
  #perfmatters
  */
  -webkit-transform: translate(-250px,0);
  transform: translate(-250px,0);
}

.c2 {
  width: 100%;
  position: absolute;
}

.c3 {
  left: 100%;
}

.c1.open {
  -webkit-transform: translate(0,0);
  transform: translate(0,0);
}

.c3.open {
  -webkit-transform: translate(-250px,0);
  transform: translate(-250px,0);
}

@media (min-width: 500px) {
  /* If the screen is wider then 500px, use Flexbox */
  .container {
    display: -webkit-flex;
    display: flex;
    -webkit-flex-flow: row nowrap;
    flex-flow: row nowrap;
  }
  .c1 {
    position: relative;
    -webkit-transition: none 0s ease-out;
    transition: none 0s ease-out;
    -webkit-transform: translate(0,0);
    transform: translate(0,0);
  }
  .c2 {
    position: static;
  }
}

@media (min-width: 800px) {
  body {
    overflow-x: auto;
  }
  .c3 {
    position: relative;
    left: auto;
    -webkit-transition: none 0s ease-out;
    transition: none 0s ease-out;
    -webkit-transform: translate(0,0);
    transform: translate(0,0);
  }
}

6.18.5. Viewport sizes of device

// @media only screen and (max-width:1920) {}
// @media all and (...) {...}

@media (max-width:1920px){}

/* iPadPro@H1366 */
@media (max-width:1400px){}

/* iPad@H1024 */
@media (max-width:1267px){}

/* iPadPro@1024 */
@media (max-width:1201px){}

@media (max-width:1023px){}

@media (max-width:967px){}

/* iPad@768, iPhone6+@H736, Nexus 6P/5X@H732 */
@media (max-width:867px){}

/* iPhone6@H667 */
@media (max-width:697px){}

/* GalaxyS5@H640 */
@media (max-width:645px){}

/* iPhone5@H568 */
@media (max-width:597px){}

@media (max-width:467px){}

/* iPhone6+@414, Nexus 6P/5X@412 */
@media (max-width:420px){}

/* iPhone6@375, GalaxyS5@360 */
@media (max-width:375px){}

/* iPhone5@320 */
@media (max-width:320px){}

Mobile first 

.any {
  width:100%;
}

@media (min-width:576px) {}
@media (min-width:768px) {}
@media (min-width:992px) {}
@media (min-width:1200px) {}

6.19. @keyframes

  • IE10+, iOS 6.1+, Android 4.4.4+
  • animation: /animation-name/ /animation-duration/ /animation-timing-function/ /animation-delay/ /animation-iteration-count/ /animation-direction/ /animation-fill-mode/ /animation-play-state/;

  • <i class="fa fa-repeat fa-spin"></i> 

    @keyframes fa-spin {
 0%   { transform: rotate(0deg); } /* this is called a step */
 100% { transform: rotate(359deg); }
 /* 0% and 100% steps share the same CSS, you can */
 /* 0%, 100% {} */
 /* Some effects have high performance animation: 
transform: translate()
transform: scale()
transform: rotate()
opacity
  */
}
.fa-spin {
 animation: fa-spin 2s linear 3s infinite alternate;
/* 
  2s :: animation-duration Xs or Xms
  3s :: animation-delay Xs or Xms. delay before start, once started, the delay has no effect
  infinite :: animation-iteration-count inifinite or int X
  alternate :: animation-direction normal,alternate

  animation-play-state :: paused, running

  animation-fill-mode:
none (default, after animation ends, styles go back to before start) | 
forwards (after animation ends, styles got applied) | 
backwards (before animation starts and during delay, apply styles defined in the 1 iteration, e.g. from and to ) | 
both;

  multiple animations can be separated by comma
*/
}
.fa-repeat:before {
  content: "\f01e";
}

  • ease (default), ease-in, ease-out, ease-in-out, linear, step-start, step-end 

    .move-me {
  display: inline-block;
  padding: 20px;
  color: white;
  position: relative;
  margin: 0 0 10px 0;
}

/* jump-term :: jump-start/start, jump-end/end, jump-none, jump-both 
   - start :: the first jump happens when the animation begins
   - end :: the last jump happens when the animation ends
*/
.move-me-1 {
  animation: move-in-steps 8s steps(4) infinite;
}
.move-me-2 {
  animation: move-in-steps 8s steps(4, start) infinite;
}
.move-me-3 {
  animation: move-in-steps 8s infinite;
}

body {
  padding: 20px;
}

@keyframes move-in-steps {
  0% {
left: 0;
background: blue;
  }
  100% {
left: 100%;
background: red;
  }
}

  • Javascript starts keyframes animation <div id="myDIV" onclick="startAnimation()">Click to start animation</div> 

    var x = document.getElementById("myDIV");
function startAnimation() {
 x.style.animation = "fa-spin 2s liner 3s infinite alternate"; // Standard
}

  • Javascript animation event listners 

    x.addEventListner("animationstart", func1);
x.addEventListner("animationiteration", func2);
x.addEventListner("animationend", func3);
  • See Animate.css

6.20. @import

/* @import usage should be avoided and should appear before any css styles defined in a css file */
@import "mystyle.css";
@import url("mystyle.css");
@import "mystyle.css" print;
/* only media types not media features */

6.21. @viewport

WD. Only Edge. Same as html:meta:viewport 

@media screen and (max-width: 400px) {
 @-ms-viewport { width: 320px; }
 /* Define the viewport size */
}

6.22. @supports css:@supports

@supports not (display:flex) {}
@supports (display: flex) or 
          (display: -ms-flexbox) or
          (display: -moz-box) {}

@supports (transition-property: color) or 
          ( (animation-name: myAnimation) and (transform: rotate(225deg) )
          ) {}

6.23. Flexbox

6.23.1. Basics

.container {
  display: flex;
  /* flex-flow: <flex-direction> <flex-wrap> */

  /* flex-direction: row | row-reverse | column | column-reverse; */
  /* main axis is by default horizontally left to right
   * main axis is changed by direcion css property (say change it ot rtl, right to left)
   * row is left to right , column is top to bottom
   *
   */
  /* flex-wrap: nowrap | wrap | wrap-reverse; */
  flex-flow: row wrap;

  /* justify-content: flex-start | flex-end | center | space-between | space-around; */
  /* e.g. you can change from flex-end to space-around for menu nav with @media queries */
  justify-content: space-around;

  /* align-items: stretch | flex-start | flex-end | center | baseline;

     This is for aligning items on one line (default x axis) on cross axis (y axis)
     relative to the same line. e.g. you have 2 lines of items and each line holds 3 items.
     3 items on each line have different height
     how do the 3 items align on y axis on every line?
     default is stretch which means all 3 items on the same line have equal height
     You have 2 options to set the items to have intrinsic height
     1. Set align-items: flex-start on the container, then every item on a line has a different height
     2. Set align-self: flex-start on an item to make that item have its intrinsic height
   */

  /* align-content: flex-start | flex-end | center | space-between | space-around | stretch;
   * This is for space between multiple lines
   * e.g. There're 3 lines of items and each line has 3 items
   * How do the 3 lines align on the y axis on each line?
   * default is stretch
   * It only works when there are multiple lines and the container height is larger than total children height
   */

  /* It's always to set width and height in container */
  width: 100%;
}

.item {
  /* order: <integer>; // optional */

  /* flex: none | <flex-grow> <flex-shrink> <flex-basis> */
  /* Default flex:0 1 auto; */

  /* flex-grow: 0 | <number>;

     Determines how the flex-item will expand if there's extra space in the container

     0 is default and means do not enlarge item in width (default x axis)
   */

  /* flex-shrink: 1 | <number>

     Determines how the flex-item will shrink if there isn't enough space in the container

     1 is default and means the item can be shrank in width (default x axis)

     Set to 0, flex-items might overflow
   */

  /* flex-basis: auto | <length>;

     Sets the initial size of the flex-item

     if it's 0, the final width of the flex-item only depends on grow/shrink factor ratio. Using auto or others, the final width is based on that set length and the amount of grow/shrink

     Set to 0% instead of 0
   */

  /* don't set width in flex item */
  flex: 0 0 100px; /* instead of width:100px */
  flex: 0 0 50%; /* width: 50% */
  /* However, flex-basis is not guaranteed and it's based on max-width and min-width */

  flex: 0 0 100%; /* use this in child item instead of width:100%, make sure parent has flex-wrap: wrap */

  flex: 1 0 auto; /* grow to the rest of the width, no shrink, auto */

  /* Override align-items in container */
  /* align-self: stretch | auto | flex-start | flex-end | center | baseline; */

}

.item_1 {
  /* Set margin:auto in an item will absorb all space of the current line in current direction
  margin-right: auto;
  /* all other items to the right of item_1 will not have extra space */
}

6.23.2. Insert a line break in flex items. Might not be easy..

.line-break {width: 100%}

<div class="container">
 <div class="item">1</div>
 <div class="item">2</div>
 <div class="line-break"></div>
 <div class="item">3</div>
 <div class="item">4</div>
</div>

6.23.3. flex-grow:1

You can flex-grow a child's width or height (depending on the flex-direction of the container), so that the child takes up the remaining width or height of the container. Just remember to set the width or height for the container.

Combine with Bootstrap div.container and div.row, you can set a specific row to take the rest of the container height. You don't have to get height of any other rows. 

.container.flex-column {
  display:flex;
  flex-direction:column;
  height: 100%; /* Make sure the container has a height */
  /* This is to take the height of the parent of the container */
}

.row.cover-row {
  flex-grow:1;
}

6.23.4. Examples

.flex-r-nw-sb {
  /* row nowrap space-between */
  display:flex;
  flex-flow: row nowrap;
  width:100%;
}
.flex-r-nw-sb-fs {
  /* row nowrap space-between don't stretch items' height on each line */
  display:flex;
  justify-content:space-between;
  flex-flow: row nowrap;
  align-items:flex-start;
  width:100%;
}

.flex-r-nw-sb-mq-r-w-c {
  /* row nowrap space-between */
  /* same as flex-r-nw-sb but with @media query */
  display:flex;

  flex-flow: row nowrap;
  width:100%;
}

@media only screen and (max-width: 597px) {
  .flex-r-nw-sb-mq-r-w-c {
    flex-flow: row wrap;
  }
}

6.24. Equal width for unknown number of child elements

<div>
   <span>tab1</span>
   <span>tab2</span>
   <span>tabx</span>
</div>

div{
  margin:0;
  padding:0;
  width:100%;
  max-width:100%;
  display:table; /* trick */
  table-layout: fixed; /* trick */
}
span{
  border:1px solid grey;
  padding 0 20;
  margin:0;
  overflow:hidden;
  text-overflow: ellipsis;
  white-space: nowrap;
  display:table-cell; /* trick */
}

/* Set back
div {
  display:block;
  table-layout:auto;
}
span {
  display:inline;
}
*/

6.25. Grid

https://css-tricks.com/snippets/css/complete-guide-grid/

6.25.1. container: grid, grid-template

grid

https://developer.mozilla.org/en-US/docs/Web/CSS/grid 

grid: <'grid-template'> | <'grid-template-rows'> / [ auto-flow && dense? ] <'grid-auto-columns'>? | [ auto-flow && dense? ] <'grid-auto-rows'>? / <'grid-template-columns'>
grid-template

https://developer.mozilla.org/en-US/docs/Web/CSS/grid-template 

.container {
  grid-template: none | subgrid | <grid-template-rows> / <grid-template-columns>;
}

/* Keyword value */
grid-template: none;

/* grid-template-rows / grid-template-columns values */
grid-template: 100px 1fr / 50px 1fr;
grid-template: auto 1fr / auto 1fr auto;
grid-template: [linename] 100px / [columnname1] 30% [columnname2] 70%;
grid-template: fit-content(100px) / fit-content(40%);

/* grid-template-areas grid-template-rows / grid-template-column values */
grid-template: "a a a"
       "b b b";
grid-template: "a a a" 20%
       "b b b" auto;
grid-template: [header-top] "a a a"     [header-bottom]
         [main-top] "b b b" 1fr [main-bottom]
            / auto 1fr auto;

/* Global values */
grid-template: inherit;
grid-template: initial;
grid-template: unset;

6.25.2. container: display, grid-template-columns, grid-template-rows

css:grid:container:grid-template-columns css:grid:container:grid-template-rows 

<!-- div.container has to be the direct parent of all grid items -->
<div class="container">
 <!-- .item are direct descendants -->
 <div class="item item-1"></div>
 <div class="item item-2">
  <div class="sub-item">
    <!-- .sub-item is not a grid item! -->
  </div>
 </div>
 <div class="item item-3"></div>
</div>

.container {

 display: grid | inline-grid | subgrid;
 /* subgrid to indicate the grid container is a grid item of another grid container and 
    it should take the parent's sizes of rows/columns  
 */

 /* Don't use column, float, clear and vertical-align in grid container as they have no effect */

 /*
 grid-template-columns: <track-size> ... | <line-name> <track-size> ...;
 grid-template-rows: <track-size> ... | <line-name> <track-size> ...;

 <track-size> - can be a length, a percentage, or a fraction of the free space in the grid (unit is fr)
 <line-name> - an arbitrary name
 */

 grid-template-columns: 40px 50px auto 50px 40px; /* 5 columns */
 grid-template-rows: 25% 100px auto; /* 3 rows */

 /* Add line names
 grid-template-columns: [first] 40px [line2] 50px [line3] auto [col4-start] 50px [five] 40px [end];
 grid-template-rows: [row1-start] 25% [row1-end row2-start] 100px [third-line] auto [last-line];

 /* Any row or column line can have multiple names: [row1-end row-start]
    Any row or column line can have the same name */

 grid-template-columns: repeat(3, 20px [col-start]) 5%;
 /* eq. to */
 grid-template-columns: 20px [col-start] 20px [col-start] 20px [col-start] 5%;

 /* Free space is defined as total space minus any fixed size items */
 grid-template-columns: 1fr 50px 1fr 1fr;
 /* each column will take (total width - 50px) * (1/3) */
}

6.25.3. container: grid-template-areas css:grid:container:grid-template-areas

.item-a{
  grid-area: header;
}
.item-b{
  grid-area: main;
}
.item-c{
  grid-area: sidebar;
}
.item-d{
  grid-area: footer;
}

.container{
  grid-template-columns: 50px 50px 50px 50px;
  grid-template-rows: auto;
  grid-template-areas: "header header header header"
           "main main . sidebar"
           "footer footer footer footer";
}
/* . means that single cell is empty. You can have multiple dots without space ".." */
/*
The first row line and column line of footer area have an extra name "footer-start"
The last row line and column line of footer aree have an extra name "footer-end"
*/

6.25.4. container: grid-column-gap, grid-row-gap, grid-gap css:grid:container:grid-gap

Gutter size: space inbetween 2 row lines or column lines. But start and end space is always 0 

grid-gap: <grid-row-gap> <grid-column-gap>;
grid-column-gap: 10px;
grid-row-gap: 15px;

6.25.5. container: justify-items, item: justify-self css:grid:container:justify-items

justify-items: start | end | center | stretch (default);
/* Adjust content in a grid item along the row axis
   start: align to left end of the grid area
   end: right
   stretch: fills the whole width of the grid area
*/

.item {
 justify-self: /* To overwrite .container: justify-items */
}

6.25.6. container: align-items, item: align-self css:grid:container:align-items

align-items: start | end | center | stretch (default);
/* Adjust content in a grid item along the column axis
   start: align content to the top of the grid area
   end: bottom
   stretch: fills the whole height of the grid area
*/

.item {
  align-self: start | end | center | stretch (default);
}

6.25.7. container: justify-content css:grid:container:justify-content

Align the grid along the row axis. If all grid items are sized with non-flexible units like px, the total size of grid might be less than the size of its grid container. Total width of all grids is less than width of its grid container. start: align all grids to the left of the grid container end: right stretch: resize the grid items to allow the grid to fill the full width of the grid container space-around: place an even amount of space between each grid item, with half-sized spaces on the far ends space-between: place an even amount of space between each grid item, with no space at the far ends space-evenly: place an even amount of space between each grid item, including the far ends 

justify-content: start | end | center | stretch | space-around | space-between | space-evenly

6.25.8. container: align-content css:grid:container:align-content

Align the grid along the column axis. Total height of all grids is less than height of its grid container. start: align all grids to the top of the grid container 

align-content: start | end | center | stretch | space-around | space-between | space-evenly

6.25.9. container: grid-auto-columns, grid-auto-rows

css:grid:container:grid-auto-columns css:grid:container:grid-auto-rows 

grid-template-columns: 60px 60px;
grid-tempalte-rows: 90px 90px;
/* 2x2 grids are created. 3 row lines and 3 column lines */
/* but item-b is out of bound */
.item-b {
 grid-column: 5 / 6; /* From the 5th column line to the 6th */
 grid-row: 2 / 3; /* From the 2nd row line to the 3rd */
}
/* It's 5x2 grids now. */
grid-auto-columns: 60px; /* the implicit grid tracks width is 60px but .item-b is auto */
grid-auto-rows: auto;

6.25.10. container: grid-auto-flow css:grid-auto-flow

For items that are not positioned (css:grid:item:grid-column, grid-row) nor put in a named area (css:grid:item:grid-area) which is later used by css:grid:container:grid-template-areas, these items are "not explicitly placed on the grid". Use this to control the auto placement 

grid-auto-flow: row | column | row dense | column dense;
/* row : fill rows first (default)
   column: fill columns first
   dense : fill smaller items first (order of items might change!)
*/

.container { /* 5x2 */
  display: grid;
  grid-template-columns: 60px 60px 60px 60px 60px;
  grid-template-rows: 30px 30px;
  grid-auto-flow: row;
}
.item-a { /* 2x1 */
  grid-column: 1;
  grid-row: 1 / 3;
}
.item-e { /* 2x1 */
  grid-column: 5;
  grid-row: 1 / 3;
}       

<section class="container">
  <div class="item-a">item-a</div>
  <div class="item-b">item-b</div>
  <div class="item-c">item-c</div>
  <div class="item-d">item-d</div>
  <div class="item-e">item-e</div>
</section>

6.25.11. item: grid-column, grid-row css:grid:item:grid-column, grid-row

css:grid:item:grid-column-start css:grid:item:grid-column-end css:grid:item:grid-row-start css:grid:item:grid-row-end

Position and resize an item 

grid-column: <grid-column-start> / <grid-column-end>;
grid-row: <grid-row-start> / <grid-row-end>;

.item-c {
  grid-column: 3 / span 2;
  grid-row: third-line / 4;
}
6.25.11.1. item: grid-column-start, grid-column-end, grid-row-start, grid-row-end

css:grid:item:grid-column-start css:grid:item:grid-column-end css:grid:item:grid-row-start css:grid:item:grid-row-end 

.item {
  grid-column-start: <number> | <name> | span <number> | span <name> | auto
  grid-column-end: <number> | <name> | span <number> | span <name> | auto
  grid-row-start: <number> | <name> | span <number> | span <name> | auto
  grid-row-end: <number> | <name> | span <number> | span <name> | auto
}

.item-a {
  grid-column-start: 2;
  grid-column-end: five; /* line-name */
  grid-row-start: row1-start;
  grid-row-end: 3;
}

.item-b {
  grid-column-start: 1;
  grid-column-end: span col4-start; /* span to a line-name */
  grid-row-start: 2;
  grid-row-end: span 2; /* span 2 rows */
}

6.25.12. item: grid-area css:grid:item:grid-area

Refer to css:grid:container:grid-template-areas 

grid-area: <name> | <grid-row-start> / <grid-column-start> / <grid-row-end> / <grid-column-end>

6.25.13. item: justify-self, align-self css:grid:item:justify-self css:grid:item:align-self

Used to overwrite grid container's css:grid:container:justify-items, css:grid:container:align-items

6.25.14. IE10, IE11

IE doesn't support: https://rachelandrew.co.uk/archives/2016/11/26/should-i-try-to-use-the-ie-implementation-of-css-grid-layout/ 

display: -ms-grid;
display: grid;
grid-gap:10px; /* not supported in IE, use grid*/
-ms-grid-columns: 100px 10px 100px 10px 100px; /* 2 extra columns are for gap */
-ms-grid-rows: 100px 10px 100px; /* the extra row is for gap */
grid-template-columns: 100px 100px 100px; /* notice there are 3 columns here while in IE there are 5 columns */

.item 

-ms-grid-row: 1;
-ms-grid-column: 1;
-ms-grid-column-span: 3;
grid-column: 1 / 3;
grid-row: 1;

6.26. Horizontal and Vertical Center

If both the parent and child are blocks, use flexbox.

For text or to place an absolute child element inside parent and the child element is not the only child. Refer to https://www.smashingmagazine.com/2013/08/absolute-horizontal-vertical-centering-css/ 

.center {
 height:200px;
 position:relative;
}
.center p {
 margin:0;
 position:absolute;
 top:50%;
 left:50%;
 transform: translate(-50%, -50%);
}

bs:Center Tags Vertically center span inside h1

Vertically align <img> and <span> inside <a> 

a {
  line-height:20px;
  display:block;
  vertical-align:middle;
}
a img {
  height: 15px; /* let's say the img's height is 15px */
}
a span {
  display:inline-block;
  vertical-align:middle;
  line-height:1em;
}

6.27. Column for text

<div>
 <h2>Title takes 2 columsn</h2>
 long text..
</div>

div {
 column-count: 3;
 column-gap: 40px;
 column-rule-style: solid; /* just like border */
 column-rule-width: 1px;
 column-rule-color: lightblue;
 column-rule: 1px solid lightblue;
}
div h2 {
 column-span: 2;
}

div.fixedWidth {
 columns: 200px; /* column-count will be auto */
 overflow-x: scroll;
 height: 600px; /* So that scrollbar shows on x axis */
}

6.28. Shape

At the end of 2016, only Chrome, Safari, iOS and Android browswers support shape. 

<div class="conainer">
  <img id="image" src="a.jpg">
  <p id="content">Long text</p>
</div>

img {
 display: block;
 float:left;
 width: 100%;
 height: auto;
 shape-outside: url(shape.png); /* polygon can be used */
 /* png has areas that are partially or completely transparent */
 shape-image-threshold: 0.9; /* The shape is any areas that have opacity greater than 0.9 */
 shape-margin: 20px; 
}

6.29. SVG

xml header should be removed when used in HTML 

<?xml version="1.0" encoding="utf-8"?>

6.29.1. Make SVG, Optimize SVG, IE 11

Exported from AI and optimize it or use nodejs:svgo

IE 11 doesn't support url('a.svg') in background CSS. Use nodejs:svgo to convert to URL encoded. 

div.svgbg {
  background-image:url('url-encoded-svg');
}

6.29.2. SVG as background image IE 11 svg:ie11

AI export .svg by default is responsive and no width and height attributes are in <svg> You may add width and height to <svg> based on viewBox 

<!-- before -->
<svg viewBox="0 0 22.77 28.96">

<!-- after -->
<svg viewBox="0 0 22.77 28.96" width="22.77" height="28.96">

If it still doesn't work, move <styles> to attributes

  • You need to export .svg in Adobe Illustrator using one of these mode for the Styling
    Presentation Attributes
    fill="none" stroke="#cfdf00" stroke-width="0.44"
    Inline Style
    style="fill:none;stroke:#cfdf00;stroke-width:0.4399999976158142px"
    (no term)
    But not Internal Style!
  • If AI already exports the .svg, use nodejs:svgo:move styles

6.29.3. SVG as background image dynamic size

You can only control size in this method 

.logo {
 background: url(logo.svg) no-repeat top left;
 background-size: contain;
 width: 100px;
 height: 82px;
}

6.29.4. CSS in SVG

Inline CSS 

<svg xmlns="http://www.w3.org/2000/svg"
    xmlns:xlink="http://www.w3.org/1999/xlink">

    <style type="text/css" >
  <![CDATA[

    circle.myGreen {
       stroke: #006600;
       fill:   #00cc00;
    }
   circle.myRed {
      stroke: #660000;
      fill:   #cc0000;
   }

  ]]>
    </style>

    <circle  class="myGreen" cx="40" cy="40"  r="24"/>
    <circle  class="myRed"   cx="40" cy="100" r="24"/>
</svg>

External CSS 

<?xml-stylesheet type="text/css" href="svg-stylesheet.css" ?>
<svg xmlns="http://www.w3.org/2000/svg"
    xmlns:xlink="http://www.w3.org/1999/xlink">
    <circle cx="40" cy="40" r="24"
       style="stroke:#006600; fill:#00cc00"/>
</svg>

Embedded CSS in SVG only affects the SVG not other elements in the global HTML if svg is not inline with global HTML.

Closed strokes forms shapes or path. Shapes or path's can be filled. And stroke (color) and stroke-width (default 1px when stroke is set) can be set.

6.29.5. SVG Zoom and Pan

<g id="parentGraphic">
 <rect />
 <text></text>
</g>

<!-- use copy an element -->
<use href="#parentGraphic" transform="translate(40, 30) scale(0.9)"></use>

<!-- Original -->
<svg currentScale="1" width="300px" height="200px" viewbox="0 0 300 200">
<!-- Don't use currentScale to zoom -->

<!-- Move down 200/200*25=25px and move right 300/300=50 px -->
<svg width="300px" height="200px" viewbox="-50 -25 300 200">

<!-- Enlarge 50% -->
<svg width="300px" height="200px" viewbox="0 0 150 100">

<!-- Enlarge 50%, move down 200/100*25=50px and move right 300/150*50=100px -->
<svg width="300px" height="200px" viewbox="-50 -25 150 100">

viewBox = <min-x> <min-y> <w> <h>

Assuming min-x and min-y is both 0 Original width and height is width and height (200x200)

50% smaller, just change w and h to w*2 and h*2

Now the circle radius is shrank from 100 to 50 You need to move down and right 50px

200/(w*2)*<min-y>=50 <min-y> = 50/200*(w*2)

<min-y> = -(target move down amount)/(svg width)*(w*(shrink factor)) <min-x> = -(target move right amount )/(svg height)*(h*(shrink factor))

6.29.6. SVG Sprite

  • Use <symbol>, and they don't display until you <use> them
<svg xmlns="http://www.w3.org/2000/svg" style="display: none;">

  <symbol id="beaker" viewBox="214.7 0 182.6 792">
    <!-- add more accessibility -->
    <title>original-file's-title</title> 
    <desc>original-file's-desc</desc>

    <!-- <path>s and whatever other shapes in here -->  
  </symbol>

  <symbol id="shape-icon-2" viewBox="0 26 100 48">
    <!-- <path>s and whatever other shapes in here -->  
  </symbol>

</svg>

<!-- We ain't even need viewBox round here. --> 
<!-- xlink:href="https://mysite.ca/a.svg#shape-icon-1"-->
<!-- xlink:href="a.svg#shape-icon-1"-->
<!-- Or the <svg><symbol></symbol></svg> is in the same document, use the following -->

<svg class="icon">
  <use xlink:href="#shape-icon-1" />
</svg>

<svg class="icon">
  <use xlink:href="#shape-icon-2" />
</svg>
About CSS styling for <svg><use>
https://tympanus.net/codrops/2015/07/16/styling-svg-use-content-css/
(no term)
Polyfil svg4everybody searches and replaces svg use with <object>

New way using <symbol> only works when <svg><use> (inline). If you want SVG-as-<img> or SVG-as-background-image, use this: 

<defs>
  <style>
    g {
  display: none;
    }
    g:target {
  display: inline;
    }
  </style>
</defs>

<g id="icon-clock">...</g>
<g id="icon-heart">...</g>
<g id="icon-arrow-right">...</g>

<div class="lili-svg-as-bg"></div>

<style>
.lili-svg-as-bg {
  background:url('//y.ca/a.svg#icon-clock') left top repeat;
  width:100%;
  height:400px;
  font-size:1rem;
}
</style>

You can also place each svg one after another 

<view id="icon-clock-view" viewBox="0 0 32 32" />
<view id="icon-heart-view" viewBox="0 32 32 32" />
<view id="icon-arrow-right-view" viewBox="0 64 32 32" />

<img src="sprite.svg#svgView(viewBox(0, 0, 32, 32))" alt="">
// or
<img src="sprite.svg#icon-heart-view" alt="">

<style>
.icon-clock {
  background: url("sprite.svg#svgView(viewBox(0, 0, 32, 32))") no-repeat;
  // or
  background: url(sprite.svg#icon-clock-view) no-repeat;
  // or
  background: url("sprite.svg") no-repeat;
  background-size: 32px 96px;
  background-position: 0 -32px;
}
</style>
6.29.6.1. Old way

Old Way uses <defs>, viewBox has to be defined in <svg> 

<svg>
  <defs>
    <g id="shape-icon-1">
  <!-- all the paths and shapes and whatnot for this icon -->
    <g>
    <g id="shape-icon-2">
  <!-- all the paths and shapes and whatnot for this icon -->
    <g>
  </defs>
</svg>

<!-- These viewBox's better be right or the icons won't look right! -->

<svg class="icon" viewBox="214.7 0 182.6 792">
  <use xlink:href="#shape-icon-1" />
</svg>

<svg class="icon" viewBox="0 26 100 48">
  <use xlink:href="#shape-icon-2" />
</svg>

6.30. Sass css:scss css:sass

6.30.1. Variables

$childMargin: 20px;
$numItems: 4 !default;

$enable-rounded: true !default;

$border-radius: .25rem !default;
$badge-border-radius: $border-radius !default;

$t_min_width: 992px;
$t_max_width: 3000px; 
$t_min_font: 46.29px;
$t_max_font: ($t_max_width * $t_min_font / $t_min_width); // 123px

.featured-media {
    display:block;
    float:left;
    font-size: #{3 * 16}px; // eq. font-size: (3 * 16px);
    font-size: $t_max_font;
    margin:0 #{$childMargin} 20px 0;
    width: calc( ( 100% - ( #{$numItems} - 1 ) * #{$childMargin} ) / #{$numItems} );
    &.every_fourth{
    margin-right:0;
    }
}

@include media-breakpoint-down(sm) {
    $numItems: 2;
    .featured-media {
    width: calc( ( 100% - ( #{$numItems} - 1 ) * #{$childMargin} ) / #{$numItems} );
    &.every_second{
        margin-right:0;
    }
    }
}

6.30.2. &

a {
    &:hover {

    }
    &.myclass {
  /* a.myclass */
    }
}

.entry {
    .single &,
    .page & {
    /* .single .entry, .page .entry */
    /* Useful for multiple parents */
    }
    &--col-1 {
    /* .entry--col-1 */
    }
}

6.30.3. @extend % sass:@extend

%message-shared {
  border: 1px solid #ccc;
  padding: 10px;
  color: #333;
}

.message {
  @extend %message-shared;
}

.success {
  @extend %message-shared;
  border-color: green;
}

/* css */
.success, .message {
  border: 1px solid #ccc;
  padding: 10px;
  color: #333;
}

.success {
  border-color: green;
}

6.30.4. Functions

  • map-keys(("foo": 1, "bar": 2)) => "foo", "bar"
  • map-get(("foo": 1, "bar": 2), "foo") => 1

6.30.5. Mixin @mixin @include

 @mixin flex {
     // write the css here
     display: -webkit-flex;
     display: flex;
 }

.row {
     @include flex;
 }

 @mixin grid($flex) {
     @if $flex {
     @include flex;
     } @else {
     display: block;
     }
 }
 @include grid(true);

 @mixin grid($flex, $full-width) {
     // multiple variables
 }

 @mixin grid($max-width, $flex: true) {
     // var with default have to be the end
 }


 @mixin padding($values...) {
     // noticie 3 dots
     @each $var in $values {
     padding: #{$var};
     }
 }
 a {
     @include padding(2px 4px 6px);
 }

 @mixin padding($values) {    
     @each $var in $values {
     padding: #{$var};
     }
 }

 // without 3 dots, it will result:
 a {
     padding: 2px;
     padding: 4px;
     padding: 6px;
 }

 $style1: 100%, 70px, #fo6d06;
 $style2: (background: #bada55, width: 100%, height: 100px);
 @mixin box($width, $height, $background) {
     width: $width;
     height: $height;
     background-color: $background;
 }
 .fogdog {
     @include box($style1...);
 }
 .badass {
     @include box($style2...);
 }

 // result:
 .fogdog {
     width: 100%;
     height: 70px;
     background-color: #fo6d06;
 }
 .badass {
     width: 100%;
     height: 100px;
     background-color: #bada55;
 }

 // use @content
 @mixin small() {
     @media only screen and (max-width: 320px) {
     @content;
     }
 }
 @include small {
     // css code for small screens go here
     width:123px;
 }

 // result
 @media only screen and (max-width: 320px) {
     width:123px;
 }

6.30.6. Bootstrap

6.30.6.1. Variables
$grid-breakpoints: (
  xs: 0,
  sm: 576px,
  md: 768px,
  lg: 992px,
  xl: 1200px
) !default;

$container-max-widths: (
  sm: 540px,
  md: 720px,
  lg: 960px,
  xl: 1140px
) !default;
$grid-gutter-width:           30px !default;

.myclass {
    $tempWidth: (map-get($container-max-widths,xl) - $grid-gutter-width )/12*6 - ($featuredSliderPadding+1)*2;
    width: $tempWidth/6 - $grid-gutter-width/2;
}
6.30.6.2. Functions
// Returns a blank string if smallest breakpoint, otherwise returns the name with a dash in front.
// Useful for making responsive utilities.
//
//    >> breakpoint-infix(xs, (xs: 0, sm: 576px, md: 768px, lg: 992px, xl: 1200px))
//    ""  (Returns a blank string)
//    >> breakpoint-infix(sm, (xs: 0, sm: 576px, md: 768px, lg: 992px, xl: 1200px))
//    "-sm"
@function breakpoint-infix($name, $breakpoints: $grid-breakpoints) {
  @return if(breakpoint-min($name, $breakpoints) == null, "", "-#{$name}");
}


// Minimum breakpoint width. Null for the smallest (first) breakpoint.
//
//    >> breakpoint-min(sm, (xs: 0, sm: 576px, md: 768px, lg: 992px, xl: 1200px))
//    576px
// sample usage $min: breakpoint-min(sm, $breakpoints)
@function breakpoint-min($name, $breakpoints: $grid-breakpoints) {
  $min: map-get($breakpoints, $name);
  @return if($min != 0, $min, null);
}
6.30.6.3. Mixins
  • media-breakpoint-up
@mixin media-breakpoint-up($name, $breakpoints: $grid-breakpoints) {
  $min: breakpoint-min($name, $breakpoints);
  @if $min {
    @media (min-width: $min) {
  @content;
    }
  } @else {
    @content;
  }
}

@each $breakpoint in map-keys($grid-breakpoints) {
  @include media-breakpoint-up($breakpoint) {
    $infix: breakpoint-infix($breakpoint, $grid-breakpoints);

    .flex#{$infix}-row            { flex-direction: row !important; }

    .justify-content#{$infix}-start   { justify-content: flex-start !important; }

  }
}

@include media-breakpoint-up(sm) {
  // ...
}

@include media-breakpoint-down(sm) {}

@include media-breakpoint-between(sm,md) {}

6.31. Use Cases

6.31.1. iOS Hover

Other than <a> tag, :hover doesn't work on iOS. Try one of these workaround 

// add onclick
<div onClick="return true" class="tast test">
    Test
</div>

// or add cursor:pointer
.test {
  background-color:blue;
}
div.test {
  cursor:pointer;
}
div.test:hover {
  background-color:red;
}

6.31.2. Image inside a div

Div's height is greater than img. add display:block in <img> 

<div class="item">
    <img src="http://placehold.it/230x20/?text=6" style="display:block">
</div>

6.31.3. Transparent Image

<img src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7"
 width="100" height="100">

6.31.4. Image scaling

IE 10+ and Edge removes any -ms-* CSS filters, so this does not work any more 

img { -ms-interpolation-mode: bicubic; }

Use this plugin https://github.com/sukhoi1/ie-bicubic-img-interpolation-plugin 

$( document ).ready(function() {
  if (navigator.appName == 'Microsoft Internet Explorer' ||  !!(navigator.userAgent.match(/Trident/) || navigator.userAgent.match(/rv:11/)) || (typeof $.browser !== "undefined" && $.browser.msie == 1))
  {
    $('img.render').bicubicImgInterpolation({
        crossOrigin: 'anonymous' //for demo purpose
    });
  }
});

6.31.5. Button Hover Double Quotes

<button class="button" style=""><span>Hover </span></button>
.button {
 display: inline-block;
 border-radius: 4px;
 background-color: #f4511e;
 border: none;
 color: #FFFFFF;
 text-align: center;
 font-size: 28px;
 padding: 20px;
 width: 200px;
 transition: all 0.5s;
 cursor: pointer;
 margin: 5px;
}
.button span {
 cursor: pointer;
 position: absolute;
 opacity: 0;
 top: 0;
 right: -20px;
 transition: 0.5s;
}
.button:hover span {
  padding-right: 25px; 
/*
push button text to left and create padding
for double quotes to push to the right of padding
*/
}
.button:hover span:after {
 opacity: 1;
 right: 0;
}

6.31.6. Text Label

Flair is a fixed font-size squared tag.

<span class="li-tag li-tag-color">Flair</span> .li-tag{ display:inline-block; margin-right: .5em; padding: 0 2px; border:1px solid; border-radius: 2px; font-size:x-small; white-space:nowrap; vertical-align:middle; } .li-tag-color{ background-color:#f5f5f5; color: #555; border-color: #ddd; } .li-tag-red{ background-color:#d9534f; color: #ffffff; border-color: #d9534f; }

Vertical align inside h1, h2 h1 .li-tag, h2 .li-tag, h3 .li-tag { margin-top: -0.5em; }

h1, h2 must have font-size defined

6.31.7. Button with icon on the left

// No space in button
<button class="li-button has-icon"><span><a href="">Youtube</a></span></button>

<style>
.li-button {
  display:inline-block;
  padding:0 8px 0 5.5px;
  border:solid 1px transparent;
  border-radius:.25em;
  background-color:#0072BC;
  color:#ffffff;
  vertical-align:middle;
  /* Overwrite default button styles */
  outline:0;
  text-decoration:none;
  cursor:none;
}
.li-button.has-icon::before {
  margin-right:6px;
  background:url(icon.svg) no-repeat;
  background-size:16px 12px;
  width:16px;
  height:12px;
  content:'';
  display:inline-block;
  vertical-align:middle;
}
.li-button span {
  vertical-align:middle;
  font-size:12px;
}
</style>

6.31.8. Vertical Line Separator in <li>

<ul class="ul">
  <li><a href="">1</a></li>
  <li><a href="">2</a></li>
</ul>

.ul li:last-child {
  margin-right: 5px;
}
.ul li:not(:first-child):before {
  content: "|";
  padding:0 5px;
}

6.31.9. Wrap h1 around a div on the right, float right sequence

<div class="div">
  <a href=""><img></a>
</div>
<h1>Long text...</h1>
.div {
  display:block;
  float:right;
}
h1 {
  clear: none; /* default is none */
}

/* Make it take the whole width */
@media (max-width: 425px) {
    .div {
        display:block;
        text-align: center;
        width:100%;
    }
}

<div id="div1" style="float:right"></div>
<div id="div2" style="float:right"></div>
<h1>...</h1>
<!-- Sequence: h1, div2, div1 -->

6.31.10. 100% width and with margin

The trick is not specify 100% width and set the margin Or 

margin-left: 15px;
/* width: 100% */
width calc(100% - 15px);

6.31.11. Align text with elements

Use flex 

<div class="flex-r-w-fs-center">
  <div class="app-information-title p-r-10 p-t-5"><span>A Title</span></div>
  <div class="app-information-btns">
    <a href="#" role="button" class="btn btn-green-overnight" target="_blank">Sign up</a>
    <a href="#" role="button" class="btn btn-green-overnight">FAQs</a>
    <a href="#" role="button" class="btn btn-green-overnight" target="_blank">Learn more</a>
  </div>
</div>

.app-information-title {
  font-size:26px;
  font-family: 'tg-medium';
  font-weight: normal;
}
.app-information-title span{
  line-height:33px;
}

6.31.12. Circle Number List

.numberCircle {
    border-radius: 50%;
    behavior: url(PIE.htc); /* remove if you don't care about IE8 */

    width: 36px;
    height: 36px;
    padding: 8px;

    background: #fff;
    border: 2px solid #666;
    color: #666;
    text-align: center;

    font: 32px Arial, sans-serif; /* 32 + padding/2 = width = height */
}

<span style="font-size: 450%; color: #FF5722;">&#9312;</span>

<span class="step">1</span>

span.step {
  background: #cccccc;
  border-radius: 0.8em;
  -moz-border-radius: 0.8em;
  -webkit-border-radius: 0.8em;
  color: #ffffff;
  display: inline-block;
  font-weight: bold;
  line-height: 1.6em;
  margin-right: 5px;
  text-align: center;
  width: 1.6em; 
}

Just change the font size to enlarge/shrink the button. Or width, line-height and the border-radius to only affect the circle (border-radius = half of the width and line-height values)

6.31.13. superscript, subscript uneven line height

<p>Some Text <sup>OM</sup>Office Mark

p {
  line-height:1.5em; /* if line-height:0 doesn't work, try increase p line-height or lower sup font-size*/
}
sup {
  /* font-size:0.5em; */
  line-height:0;
}

6.31.14. Swap image in @media

css 

.flex-r-w-fs-center-mq-r-w-sa-center {
  display:flex;
  flex-flow:row wrap;
  justify-content:flex-start;
  align-items:center;
}

.footer-logos > div {
  margin:15px 10px;
  background-color:red;
}
.footer-logos > div > div > img{
  display:block;
  margin:0 auto;
}

@media only screen and (max-width: 697px) {
  .footer-logos.flex-r-w-fs-center-mq-r-w-sa-center {
    justify-content:space-around;
  }
}

@media only screen and (max-width: 320px) {
  .footer-logos > div {
    width:100%;
    margin-top:2em;
  }

  .footer-logos .cs-i-1 {
    background:transparent url('http://placehold.it/137x42/?text=137x42') center center no-repeat;
    height:42px;
  }
  .footer-logos .cs-i-2 {
    background:transparent url('http://placehold.it/137x49/?text=137x49') center center no-repeat;
    height:49px;
  }
  .footer-logos .cs-i-3 {
    background:transparent url('http://placehold.it/137x64/?text=137x64') center center no-repeat;
    height:64px;
  }
  .footer-logos .cs-i-4 {
    background:transparent url('http://placehold.it/137x37/?text=137x37') center center no-repeat;
    height:37px;
  }
  .footer-logos .cs-items {
    margin:0.5em auto;
    width:137px;
  }
  .footer-logos .cs-items img {
    visibility:hidden;
  }

}

<div class="footer-logos flex-r-w-fs-center-mq-r-w-sa-center">
  <div>
    <div class="cs-items cs-i-1">
      <img src="http://placehold.it/81x25/?text=81x25" alt="#1 Logo"/>
    </div>
  </div>
  <div>
    <div class="cs-items cs-i-2">
      <img src="http://placehold.it/92x25/?text=92x25" alt="#2 Logo"/>
    </div>
  </div>
  <div>
    <div class="cs-items cs-i-3">
      <img src="http://placehold.it/54x25/?text=86x25" alt="#3 Hydro"/>
    </div>
  </div>
  <div>
    <div class="cs-items cs-i-4">
      <img src="http://placehold.it/70x25/?text=70x25" alt="#4 Logo"/>
    </div>
  </div>
</div>

6.31.15. Hightlight element when the hash and id of element match

:target {
  animation: contenthighlight 1s ease;
}
@keyframes contenthighlight {
  0% { border-left-color: red; }
  100% { border-left-color: white; }
}

6.31.16. Go to anchor element with position:fixed header

:target:before {
content:"";
display:block;
height:90px; /* fixed header height*/
margin:-90px 0 0; /* negative fixed header height */
}

6.31.17. Center position:absolute

Don't have to know the child's exact width 

.parent {
  position:relative;
}

/* center both horizontally and vertically
.child {
  position: absolute;
  top: 50%;  /* position the top  edge of the element at the middle of the parent */
  left: 50%; /* position the left edge of the element at the middle of the parent */

  transform: translate(-50%, -50%); /* This is a shorthand of
                   translateX(-50%) and translateY(-50%) */
}

/* center only horizontally */
.child {
  position: absolute;
  left: 50%;
  transform: translateX(-50%);
}

6.32. BEM

.Block__Element--Modifier
Block, Element, Modifier
.sky
parent block, standalone component
.sky__clouds
clouds element inside parent block sky
.sky .sky--dusk
a different sky. Modifier is dusk
// Block
.scenery {
   //Elements
  &__sky {
    fill: screen;
  }
  &__ground {
    float: bottom; 
  }
  &__people {
    float: center;
  }
}

//Block
.sky {
  background: dusk;

  // Elements

  &__clouds {
    type: distant;
  }

  &__sun {
    strength: .025;
  }

  // Modifiers
  &--dusk {
    background: dusk;
    .sky__clouds {
  type: distant;
    }
    .sky__sun {
  strength: .025;
    }
  }

  &--daytime {
    background: daylight;
    .sky__clouds {
  type: fluffy;
  float: center;
    }
    .sky__sun {
  strength: .7;
  align: center;
  float: top;
    }
  }
}

6.33. Houdini

6.33.1. Properties and Values API

@property --colorPrimary {
  syntax: '';
  initial-value: magenta;
  inherits: false;
}

.card {
  background-color: var(--colorPrimary); /* magenta */
}

.highlight-card {
  --colorPrimary: yellow;
  background-color: var(--colorPrimary); /* yellow */
}

.another-card {
  --colorPrimary: 23;
  background-color: var(--colorPrimary);
  /*
     @property fallback to magenta.
     Without @property, --colorPrimary = 23 and it's not valid for background-color and hence the browser ignores the invalid value
   */
}

6.34. CSS Frameworks

6.34.1. Bootstrap

6.34.1.1. Mobile Meta Setup
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">

<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">

<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<!-- Bootstrap css and js files are about 170kb -->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" 
  integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" 
  crossorigin="anonymous">

</head>
<body>
<!-- HTML Code -->
<script
  src="https://code.jquery.com/jquery-1.12.4.min.js"
  integrity="sha256-ZosEbRLbNQzLpnKIkEdrPv7lOy9C27hHQ+Xp8a4MxAQ="
  crossorigin="anonymous"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" 
  integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" 
  crossorigin="anonymous"></script>
</body>
</html>
6.34.1.2. Global
  • box-sizing is border-box including *:before and *:after
  • font-size:16px is declared on <html> and font-size:1rem on <body>
  • BS3 default font-size is 14px
  • <body> also sets global font-family and line-height
  • <body> has background-color #fff
  • bootstrap.js requries jQuery
  • By default, flexbox is not enabled
  • If flexbox is enabled:
    • entire grid system switches from float to flex
    • input groups move from table to flex
    • media components move from table to flex
    • After flex is enabled, you might have problems with IE9 and IE10

BS4 vs BS3

Hands on different versions of Bootstrap

6.34.1.3. Content
  1. Native font stack

    Switched from Helvetica Neue, Helvetica and Arial to:

    • -apply-system (Safari for OS X and iOS)
    • BlinkMacSystemFont (Chrome for OS X)
    • Segoe UI (Windows)
    • Robot (Android)
    • Helvetica Neue, Arial, sans-serif !default;
  2. h1, h2 heading elements
    heading elements
    margin-bottom: .5rem, no margin-top
    .h1 .h2
    to imitate heading display without margins
    (no term)
    Add .display-1, …, .display-4 to change the size and display for <h1>, <h2>, etc.
    (no term)
    Add <small class="text-muted">secondary heading text</small> to <h1>
    • inline with text in <h1>, light color
  3. <p>
    • margin-bottom: 1rem, no margin-top
    • Add .lead to <p> to make it more standout.
  4. Lists <ul> <ol> <dl>
    • margin-bottom: 1rem, no margin-top
    • Except nested lists
    • add .list-unstyled to <ul>, <ol>. Only only apply for Level-1 children <li>
    • add .list-inline to <ul, <ol> and .list-inline-item to li
    • add .dl-horizontal to <dl>
  5. Horinzontal Center (text, truncate, block)
    • Add .text-center in column
    • Add .center-block in column (margin:0 auto)
    • Add .text-truncate in column can truncate text and provides ellipsis
  6. Special Text
    <kbd> User input
    <kbd>cd</kbd>, <kbd><kbd>ctrl</kbd> + <kbd>,</kbd></kbd>
    (no term)
    <pre> has margin-top removed and add margin-bottom: 1rem;
    (no term)
    add .pre-scrollable to <pre> to set a max-height and make it horizontally scrollable
    <code>&lt;em&gt;</code>
    red text
    (no term)
    <samp>&lt;em&gt;</samp>
    (no term)
    add .small to imitate <small> (normally it's 85% of parent font-size)
    (no term)
    <mark> light yellow
    (no term)
    <s>, <del> strike
    (no term)
    <ins> same as <u> underline
    (no term)
    <em>, <i> and <var> look the same
    (no term)
    add .initialism to cap and make font size 90% <abbr title="full term">HTML</abbr>
    (no term)
    add .blockquote-reverse to <blockquote><p>...</p><footer>...</footer></blockquote>
  7. Vertical Align
    • Only applies to inline, inline-block, inline-table and table cell
    • .align-baseline, .align-top, .align-middle, .align-bottom
    • .align-text-bottom, .align-text-top
  8. Table
    • Add class="table" to <table>
      • add .table-inverse (dark) to inverse color
      • add .table-striped to add color to even or odd row within <tbody>
      • add .table-bordered
      • add .table-hover
      • add .table-sm (or .table-condensed) to remove cell padding
      • add .table-reflow to transpose
    • Add scope="row" to the first td (or th) in each <tr> inside <tbody>
    • Add class="thead-inverse" or thead-default in <thead> to make table header dark or light
    • Add colors to rows or individual cells. If table-inverse, use bg-* or text-*
      .table-active (.active)
      light grey
      .table-success (.success)
      light green
      .table-info (.info)
      light blue
      .table-warning (.warning)
      light yellow
      .table-danger (.danger)
      light red
    • Wrap table.table inside div.table-responsive to make table horizontal scrollable on small devices (under 768px)
  9. Image

    .img-responsive BS3 .img-fluid BS4 

    <!-- block, max-width: 100%; height: auto; -->
<img src="" class="img-responsive">

<!-- add .center-block to <img> to center .img-responsive -->

<!-- Or center an image as inline -->
<img src="" class="img-responsive img-circle" style="display:inline">

    max-width and max-height

    <div class="img-container" style="width:300px;height:168px;"> <img src="…"> </div> .img-container img { max-width: 100%; max-height: 100%; }

    Image shapes :: img-circle, img-rounded, img-thumbnail

  10. Colors

    text-muted, text-primary, -success, -info, -warning, -danger btn-primary, -success, -info, -warning, -danger bg-primary, -success, -info, -warning, -danger

  11. Quick float content .pull-left, .pull-right

    Not to be used in .navbar. Use .navbar-left or .navbar-right instead.

6.34.1.4. Layout
  1. Grid
    <div class="container">
  <div class="row">
    <div class="col-*-*"></div>
  </div>
  <div class="row">
    <div class="col-*-*"></div>
    <div class="col-*-*"></div>
    <div class="col-*-*"></div>
  </div>
  <div class="row">
    ...
  </div>
</div>

    For all sizes, 15px on each side of a column .container has a fixed max-width based on different viewports. .container-fluid takes 100% of the viewport width and the same as .container .row is a horizontal group of columns. Only columns can be immediate children of rows.

    15px on each side of a column

    BS4          
      <576px >=576px >=768px >=992px >=1200px
    Class .col- .col-sm- .col-md- .col-lg- .col-xg-
    container none/auto 540px 720px 960px 1140px
    max width          
               
    BS3          
        <768px >=768px >=992px >=1200px
    Class   .col-xs- .col-sm- .col-md- .col-lg-
    container   none/auto 750px 970px 1170px
    max width          

    @media (min-width: 576px) {} => >=576 @media (max-width: 576px) {} => <=576

    col-lg-9
    non large size will have 100% width
    col-md-7 col-lg-9
    small size and extra small will have 100% width
    col-xs-9 col-md-7
    large size will have 7 columns, small size will have 9 columns
    728px
    requires (BS4) md-12, lg-9, xg-8; (BS3) sm-12, md-9, lg-8
    300px
    requries (BS4) md-5, lg-4, xg-3; (BS3) sm-5, md-4, lg-3

    When column has 100% width, margin-bottom is 25px

    Prevent strange wrapping with uneven height content at certain viewport size
    div.clearfix.visible-*-(block, inline-block)

    Offset Columns

    col-md-offset-*
    increase the left margin of a column by n columns

    Offset can be used to horizontally center a column div.col-xs-offset-3.col-xs-6 :: to the left 3 columns, width 6 columns, to the right 3 columns .col-xs-offset-3 carries forward to bigger size. In order to remove offset for larger sizes, add .col-sm-offset-0

    Change Column Ordering col-sm-push-*, col-sm-pull-* :: col-sm-push-8, col-sm-pull-4. Only at sm the columns are pushed/pulled. This only works when elements can be floated in one row. When they are stacked, this trick doesn't work.

  2. Responsive Utility bs:responsive utility
    .hidden-*-down
    e.g. .hidden-md-down hides an element on xs, sm and md viewports
    .hidden-*-up
    .hidden-md-up hides an element on md, lg and xg viewports
    (no term)
    combine .hidden-*-down and .hidden-*-up to only show element for a specific viewport size
    .hidden-*
    * is xs, sm, lg, xl
    .hidden-print
    hide element for print device
    .visible-print-block
    only visible in print as display: block
    .visible-print-inline
    only visible in print as display: inline
    .visible-print-inline-block
    only visible in print as display: inline-block
    (no term)
    .visible-*-block, -inline, -inline-block where * is xs, sm, lg, xl
  3. Media Object

    Align media (image, video, audio) to the left or right of a content block 

    <div class="media">
  <!-- .media-middle, .media-bottom, default is top -->
  <!-- .media-left, .media-right -->
  <a class="media-left" href="#">
    <img class="media-object" src="..." alt="...">
  </a>
  <div class="media-body">
    <h4 class="media-heading">Media Heading</h4>
    Some Description...
    <!-- Nest another media object -->
  </div>
</div>

    Put multiple li.media inside ul.media-list. Good for layout comments

6.34.1.5. Utilities
  1. Spacing Utility (Boostrap 4)
    m
    margin
    p
    padding
    t, b, l, r
    top, bottom, left, right
    x
    *-left and *-right
    y
    *-top and *-bottom
    a
    all 4 sides
    mt-0
    margin top 0
    mx-auto
    same as .center-block

    In Bootstrap 3, use .form-group to create margin-bottom: 15px

  2. Responsive Utility

    bs:responsive utility

  3. Responsive helpers (embed and float) bootstrap:responsive helpers

    Apply to <iframe>, <embed>, <video> and <object> to maintain media size ratio 

    <div class="embed-responsive embed-responsive-16by9">
  <iframe class="embed-responsive-item" src="//www.youtube.com/embed/zpOULjyy-n8?rel=0" allowfullscreen></iframe>
</div>

    BS3 4by3 and 16by9. BS4 added 21by9 and 1by1

    Control floats based on viewport size breakpoint float-xs-left, float-xs-right, float-xs-none

    Raw CSS 

    <div class="videoWrapper-16-9">
  <!-- Copy & Pasted from YouTube -->
  <iframe width="560" height="349" src="http://www.youtube.com/embed/n_dZNLr2cME?rel=0&hd=1" style="border:0" allowfullscreen></iframe>
</div>

    .videoWrapper-16-9 {
  position: relative;
  padding-bottom: 56.25%; /* 16:9 */
  padding-top: 25px;
  height: 0;
}
.videoWrapper-16-9 iframe {
  position: absolute;
  top: 0;
  left: 0;
  width: 100%;
  height: 100%;
}
  4. Border (BS4)
    • .rounded
    • .rounded-circle
    • .rounded-top, right, left, bottom
  5. Text bs:text align
    BS4
    .text-*-left, .text-*-right, .text-*-center, * is xs, sm, md, lg, xl
    BS3
    .text-justify, .text-left, right, center
    Text nowrap
    .text-nowrap
    (no term)
    Text color
    • .text-muted
    • .text-primary, -success, -info, …
    (no term)
    Font weight
    • .font-weight-bold, .font-weight-normal
    • .font-italic
6.34.1.6. Components
  1. Alert

    Wrap text in div with .alert and one of the alert color class

    • alert-success
    • alert-info
    • alert-warning
    • alert-danger

    100% width block with inner and outter spacing

    Non-dismissible 

    <div class="alert alert-danger" role="alert">
  Some text with
  <a href="#" class="alert-link"> a link </a> 
  some other text
</div>

    Dismissable 

    <div class="alert alert-warning alert-dismissible" role="alert">
  <button type="button" class="close" data-dismiss="alert" aria-label="Close">
    <span aria-hidden="true">&times;</span>
  </button>
  <strong>Warning!</strong> Better check yourself, you're not looking too good.
</div>
  2. Jumbotron

    .page-header :: Extra space on top and some on bottom 

    <div class="page-header">
  <h1>Example page header <small>Subtext for header</small></h1>
</div>

    Jumbotron :: 100% width BS4 uses h.display-3, p.lead 

    <div class="jumbotron">
  <h1 class="display-3">Hello, world!</h1>
  <p class="lead">Text.</p>
  <hr class="my-2">
  <p>Another paragraph.</p>
  <p class="lead">
    <a class="btn btn-primary btn-lg" href="#" role="button">Learn more</a>
  </p>
</div>
  3. Tag (BS4) Label (BS3)

    bs:tag Used in <button>, <h1> as inline element and in bs:list group <span class="tag tag-default">New</span> <span class="label label-info">New</span>

    tag-pill or badge in BS3 can collapse when there's no content <span class="tag tag-pill tag-default">Default</span>

    bs:Center Tags Vertically center span inside h1 h1,h2,h3 { vertical-align:middle; }

    h1>.tag, h2>.tag, h3>.tag { vertical-aign:middle; margin-top:-0.5em; font-size: 50%; * default is 75%. Make the tag smaller * }

  4. Progress

    Contextual classes :: progress-success, etc. .progress-striped

    BS4 

    <div class="text-xs-center" id="example-caption-4">Some Text appears on top progress bar</div>
<progress class="progress" value="75" max="100" aria-describedby="example-caption-4"></progress>

    BS3 is more reliable..

    Contextual classes :: progress-bar-success, etc. .progress-bar-striped Add .active to .progress-bar-striped to animate the stripes right to left 

    <div class="progress">
  <div class="progress-bar"
       role="progressbar"
       aria-valuenow="0"
       aria-valuemin="0"
       aria-valuemax="100"
       style="min-width: 2em;">
    0%
  </div>
</div>
<div class="progress">
  <div class="progress-bar"
       role="progressbar"
       aria-valuenow="2"
       aria-valuemin="0"
       aria-valuemax="100"
       style="min-width: 2em; width: 2%;">
    2%
  </div>
</div>

<!-- Align multiple .progress-bar's on one line in ratio -->
<div class="progress">
  <div class="progress-bar progress-bar-success" style="width: 35%">
    <span class="sr-only">35% Complete (success)</span>
  </div>
  <div class="progress-bar progress-bar-warning progress-bar-striped" style="width: 20%">
    <span class="sr-only">20% Complete (warning)</span>
  </div>
  <div class="progress-bar progress-bar-danger" style="width: 10%">
    <span class="sr-only">10% Complete (danger)</span>
  </div>
</div>
  5. Button, Button Group bs:button

    Used in bs:nav bs:navbar Add .btn.btn-default to <a>, <button> and <input> Only button.btn can be used as button in nav

    btn-default, btn-primary, btn-success, info, warning, danger

    btn-lg, btn-sm, btn-xs

    btn-block :: full width of parent

    Add .active to .btn to show the button is pressed

    Button with icon 

    <a href="#" class="btn btn-default btn-lg">
 <span class="glyphicon glyphicon-search"></span> Button Large
</a>

    bs:btn-group Use role="group" for .btn-group and .btn-group-vertical div.btn-group to align <button>s horizontally without padding nor margin div.btn-group-vertical to align <button>s vertically without padding nor margin

    Use role="toolbar" for .btn-toolbar When there're multiple div.btn-group or div.btn-group-vertical, use div.btn-toolbar to wrap them.

    To group a bs:dropdown with <button>s, use div.btn-group to wrap the dropdown all together.

  6. Form

    bs:form

    .sr-only
    add to <label> to hide it
    .form-group
    add to <div> which wraps each form field to add margin-bottom
    .form-group-lg, .form-group-sm
    sizing
    1. .form-control
      .form-control
      add to <input>, <select>, <textarea> to make it width 100%
      .form-control-file
      add to <input type=file> for 100% width
      input.form-control-lg, input.form-control-sm
      sizing (BS4)
      input.input-lg, input.input-sm
      sizing (BS3)
    2. Help Text (after <input>)
      .form-text (.help-block)
      block-level help text. p.form-text or div.form-text
      .text-muted
      inline help text small.text-muted or span.text-muted
      (no term)
      aria-describedby attribute (may also refer to the feedback)
      <div class="form-group">
  <label for="inputPassword">Password</label>
  <input type="password" id="inputPassword" class="form-control" aria-describedby="passwordHelpInline">
  <small id="passwordHelpInline" class="text-muted">
    Must be 8-20 characters long.
  </small>
</div>
    3. .form-inline
      • By default, <form> doesn't have to have any class
      • add to <form> and any form field to make it inline. Only works for md and above viewport sizes
    4. Checkbox and Radio
      • div.checkbox (.form-check BS4), div.radio (BS3 only)
      • label.checkbox-inline (.form-check-inline BS4), label.radio-inline (BS3 only)
      • label.form-check-label (BS4 only)
      • input.form-check-input (BS4 only)

      <div class="checkbox"> <label> <input id="a" type="checkbox"> Only one checkbox </label> </div>

      <div class="form-group"> <label class="checkbox-inline"> <input name="a" id="a1" type="checkbox"> Checkbox #1 </label> <label class="checkbox-inline"> <input name="a" id="a2" type="checkbox"> Checkbox #2 </label> </div>

    5. .input-group bs:form:input-group

      Group multiple inputs of different types (or even text) in one line. Good for making very short form 

      <div class="form-group">
  <label class="sr-only" for="exampleInputAmount">Amount (in dollars)</label>
  <div class="input-group">
    <div class="input-group-addon">
      <input type="checkbox" checked>
    </div>
    <input type="text" class="form-control" id="exampleInputAmount" placeholder="Amount">
    <div class="input-group-addon">.00</div>
  </div>
</div>

<div class="form-group">
  <div class="input-group">
    <input type="text" class="form-control" placeholder="Search terms">
    <span class="input-group-btn">
      <button class="btn btn-default" type="submit">Go!</button>
    </span>
  </div>
</div>
    6. Align fields in columns

      Add .col-*-* to <label> and container of form field. Also add .control-label to <label> 

      <div class="form-group">
  <label class="col-sm-2 control-label" for="inputComments">Comments</label>
  <div class="col-sm-10">
    <textarea class="form-control" id="inputComments"></textarea>
  </div>
</div>

      <!-- Align Submit button -->
<div class="form-group">
  <div class="col-sm-10 col-sm-offset-2">
    <input type="submit" class="btn btn-default" value="submit">
  </div>
</div>

<!-- Align checkbox/radio -->
<div class="form-group">
 <div class="col-sm-10 col-sm-offset-2">
   <label class="checkbox-inline">
  <input name="a" type="checkbox">
  Checkbox #2
   </label>
 </div>
</div>
    7. Validation and Feedback

      Add validation class to div.form-group to add color as a whole

      • .has-success
      • .has-warning
      • .has-error
      • .has-feedback

      Validation feedback with icon and/or text (only input type=text)

      .has-feedback
      add to div.form-group
      .control-label
      add to <label>
      span.glyphicon.glyphicon-ok.form-control-feedback
      icon inside field (BS3)
      input.form-control-warning, -success, -danger
      icon inside field (BS4)
      div.form-control-feedback
      feedback in text
      <div class="form-group has-feedback">
  <label class="control-label" for="inputName">Name</label>
  <input class="form-control" id="inputName" type="text" placeholder="Name">
  <span class="glyphicon glyphicon-ok form-control-feedback"></span>
</div>
  7. Thumbnail (BS3)

    div.thumbnail is a block with padding 4px and border 1px. Place it inside a column for layout

  8. List group bs:list group

    Used in bs:panel bs:list group

    • Parent and child can be any HTML elements.
    • <ul> <li> will not have hovering effect
    • Add contextual classes to li.list-group-item
      • list-group-item-success, info, warning, danger
    <div class="list-group">
  <!-- may use .disabled, .active -->
  <li class="list-group-item disabled">
    <!-- may use bs:tag -->
    New Mails
    <span class="tag tag-default tag-pill float-xs-right">14</span>
  </li>

  <!-- BS4: link as a button, use <div> as parent -->
  <a class="list-group-item list-group-item-action">
    Dapibus ac facilisis in
  </a>

  <!-- BS3: link as a button, use <div> as parent -->
  <button type="button" class="list-group-item">
    Dapibus ac facilisis in
  </button>

  <!-- Custom Content BS3 -->
  <a href="#" class="list-group-item">
    <h4 class="list-group-item-heading">Heading</h4>
    <p class="list-group-item-text">...</p>
  </a>

  <!-- Custom Content BS4 -->
  <a href="#" class="list-group-item list-group-item-action">
    <h4 class="list-group-item-heading">Heading</h4>
    <p class="list-group-item-text">...</p>
  </a>

</div>
  9. Panel (BS3 only) bs:panel

    3 rows layout 100% width with padding

    Contextual classes: panel-success, … 

    <div class="panel panel-default">

  <div class="panel-heading">
    <h2 class="panel-title">Title</h2>
  </div>

  <div class="panel-body">
    <p>Panel Content</p>
  </div>

  <!-- may add any HTML or BS components -->

  <div class="panel-footer">
    <a href="#">Some Action</a>
  </div>

</div>

    .panel-collapse

    • Expand and collapse a panel bs:collapse
    • Place .panel-body inside .panel-collpse
    • .list-group can replace .panel-body inside .panel-collapse
    <!-- add role="tablist" and aria-multiselectable -->
<div class="panel-group"
     id="accordion"
     role="tablist"
     aria-multiselectable="true">

  <div class="panel panel-default">
    <!-- add role="tab" and id for each panel-heading -->
    <div class="panel-heading"
     role="tab"
     id="headingOne">
  <h4 class="panel-title">
    <!-- For each .panel-title
         - add regular bs:collapse action button with data-parent
    -->

    <!-- This panel opens initially -->
    <!-- no .collapsed in <a> means it expands initially -->
    <!-- aria-expanded="true" not false! -->
    <a role="button"
       data-toggle="collapse"
       data-parent="#accordion"
       href="#collapseOne"
       aria-expanded="true"
       aria-controls="collapseOne">
      Collapsible Group Item #1
    </a>
  </h4>
    </div>

    <!-- .collapse.in :: displays content initially -->
    <div id="collapseOne"
     class="panel-collapse collapse in"
     role="tabpanel"
     aria-labelledby="headingOne">
  <div class="panel-body">
    ...
  </div>
    </div>

  </div>

  <div class="panel panel-default">
    <div class="panel-heading" role="tab" id="headingTwo">
  <h4 class="panel-title">
    <!-- .collapsed :: collapse content initially -->
    <a class="collapsed"
       role="button"
       data-toggle="collapse"
       data-parent="#accordion"
       href="#collapseTwo"
       aria-expanded="false"
       aria-controls="collapseTwo">
      Collapsible Group Item #2
    </a>
  </h4>
    </div>

    <!-- .collapse :: hides content initially -->
    <div id="collapseTwo"
     class="panel-collapse collapse"
     role="tabpanel"
     aria-labelledby="headingTwo">
  <div class="panel-body">
    ...
  </div>
    </div>
  </div>

</div>
  10. Card, Card Group, Card Deck (BS4) bs:card

    Panel is replaced by Card in BS4 

    <!-- add .card-inverse to make text color white.
     contextual classes :: card-primary, info, etc.
     outline color :: card-outline-primary, info, etc.
-->
<div class="card">

  <!-- .card-header in <h*> -->
  <h3 class="card-header">Featured</h3>
  <!--  header can have bs:nav, tabs or pills -->
  <!-- just add .card-header-tabs --->
  <div class="card-header">
    <ul class="nav nav-tabs card-header-tabs">
      ...
    </ul>
  </div>

  <!-- .card-img without -top or -bottom centers the image inside .card -->
  <img class="card-img-top" src="..." alt="Card image cap">
  <!-- Overlay content on image -->
  <div class="card-img-overlay">
    <h4 class="card-title">Image title</h4>
    <p class="card-text">...</p>
    <p class="card-text"><small class="text-muted">...</small></p>
  </div>

  <!-- Wrap text with .card-block -->
  <div class="card-block">
    <h4 class="card-title">Card title</h4>
    <p class="card-text">...</p>
    <!-- regular button -->
    <a href="#" class="btn btn-primary">Go</a>
  </div>

  <div class="card-block">
    <!-- .card-title and .card-subtitle in <h*> -->
    <h4 class="card-title">Card title</h4>
    <h6 class="card-subtitle text-muted">Support card subtitle</h6>
  </div>

  <!-- bs:list group -->
  <ul class="list-group list-group-flush">
    <li class="list-group-item">Cras justo odio</li>
    <li class="list-group-item">Dapibus ac facilisis in</li>
    <li class="list-group-item">Vestibulum at eros</li>
  </ul>

  <div class="card-block">
    <!-- .card-link -->
    <a href="#" class="card-link">Card link</a>
    <a href="#" class="card-link">Another link</a>
  </div>

  <img class="card-img-bottom" src="..." alt="Card image cap">

  <div class="card-footer">2 days ago</div>
  <!-- .card-header in <h*> -->
  <h3 class="card-footer">2 days ago</h3>

</div>

    Card Group

    • wrap all .card inside div.card-group
    • each .card will have equal height with no spacing like in a table

    Card Deck

    • Like .card-group but with spacing
    • wrap all .card inside div.card-deck
    • if flex is not enabled, wrap div.card-deck with div.card-deck-wrapper

    Card Columns

    • Masonry
    • Wrap all .card inside div.card-columns
  11. Dropdown (js)

    bs:dropdown For navigation purpose. Used in bs:nav bs:navbar bs:btn-group 

    <!-- .open :: dropdown state is always open -->
<div class="dropdown open">
  <!-- .dropdown-toggle turn something into a dropdown toggle -->
  <!-- data-toggle :: attach js dropdown event -->
  <button
    type="button" id="dropdownMenuButton"
    class="btn btn-secondary dropdown-toggle"
    data-toggle="dropdown"
    aria-haspopup="true"
    aria-expanded="false">
    Dropdown button <span class="caret"></span>
    <!-- BS4 does't have to define caret -->
  </button>

  <!-- BS4 no longer requires menu items to be <li><a /></li> but you have to use .dropdown-item -->
  <!-- <ul> <li> <a> are used in BS3 -->
  <div class="dropdown-menu" aria-labelledby="dropdownMenuButton">
    <h6 class="dropdown-header">Combo A (not selectable)</h6>
    <a class="dropdown-item" href="#">1</a>
    <a class="dropdown-item" href="#">2</a>
    <a class="dropdown-item" href="#">3</a>
    <!-- BS3 use .divider -->
    <div role="separator" class="dropdown-divider"></div>
    <h6 class="dropdown-header">Combo B (not selectable)</h6>
    <a class="dropdown-item" href="#">4</a>
    <!-- disabled -->
    <a class="dropdown-item disabled" href="#">5 (Sold out!)</a>
    <a class="dropdown-item" href="#">6</a>
  </div>
</div>

    To group a bs:dropdown with <button>s, use div.btn-group to wrap the dropdown all together. 

    <div class="btn-group" role="group" aria-label="...">
  <button type="button" class="btn btn-info">Button #1</button>
  <button type="button" class="btn btn-info">Button #2</button>

  <!-- dropdown starts here -->
  <div class="btn-group" role="group">
    <button
  type="button"
  class="btn btn-info dropdown-toggle"
  data-toggle="dropdown"
  aria-haspopup="true"
  aria-expanded="false">
  Button #3: Dropdown
  <!-- BS4 doesn't need to manually add a caret -->
  <!-- <span class="caret"></span> -->
    </button>

    <!-- add ul.dropdown-menu mentioned above -->
  </div>

</div>
  12. Navigation (js), tab (js)
    1. .nav, .nav-tabs, .nav-pills, tablist (js)

      bs:nav :: May use

      Items are blocks float to left or stacked vertically 

      <!-- nav-tabs or nav-pills -->
<!-- .nav-stacked :: add to ul -->
<!-- .nav-justified :: add to ul. Center tabs/pills -->

<ul class="nav nav-tabs">

  <!-- BS4 for <li> :: add .nav-item and remove role="presentation" -->
  <li role="presentation" class="active">
    <!-- BS4 for <a> :: add .nav-link -->
    <a href="#">Item 1</a>
  </li>
  <li role="presentation" class="disabled"><a href="#">Item 2</a></li>
  <li role="presentation"><a href="#">Item 3</a></li>

  <!-- Item can be a bs:dropdown -->
  <li role="presentation" class="dropdown">
    <button
      class="dropdown-toggle"
      data-toggle="dropdown"
      aria-haspopup="true"
      aria-expanded="false">
      Item 4
    </button>

    <ul class="dropdown-menu">
      ...
    </ul>
  </li>

</ul>

      bs:tab Display content when nav-tabs or nav-pills is clicked $().tab data-toggle="tab" 

      <!-- add role="tablist" -->
<ul class="nav nav-tabs"
    role="tablist"
    id="myTab">
  <!-- Be sure to add .active -->
  <!-- BS4 for <li> :: add .nav-item and remove role="presentation" -->
  <li role="presentation" class="active">
    <!-- <a> ::
         add role="tab", data-toggle="tab", aria-controls="..."
         BS4 :: add .nav-link
         may use data-target instead of href
     -->
    <a href="#item1"
       data-toggle="tab"
       role="tab"
       aria-controls="item1">Item 1</a>
  </li>


  <li role="presentation" class="active">
    <!-- <a> ::
         add role="tab", data-toggle="tab", aria-controls="..."
         BS4 :: add .nav-link
     -->
    <a href="#item1"
       data-toggle="tab"
       role="tab"
       aria-controls="item2">Item 2</a>
  </li>

</ul>

<div class="tab-content">
  <!-- effect :: .fade to div.tab-pane -->
  <div class="tab-pane active"
       id="item1"
       role="tabpanel">
    Item 1 content
  </div>

  <div class="tab-pane active"
       id="item2"
       role="tabpanel">
    Item 2 content
  </div>
</div>
    2. Navbar (js)

      bs:navbar is responsive Use bs:collapse

      div.collapse collpses at viewport <768px which is xs in BS3 and sm in BS4

      Background image to navbar 

      @media (min-width:768px) {
  .navbar {
    background:url('../images/nav-bg.jpg') center repeat-x #222;
    background-size:auto 100%;
  }
}

@media (max-width:768px) {
  .navbar .navbar-header{
    background:url('../images/nav-bg.jpg') center repeat-x #222;
    background-size:auto 100%;
  }
}

      <!-- .navbar-fixed-bottom | .navbar-fixed-top :: add to <nav> if navbar is fixed -->
<!-- .navbar-default | .navbar-inverse (black and white) -->
<!-- customize color, border using .navbar-default -->
<nav class="navbar navbar-default">
  <!-- Without div.container, the navbar will align to left -->
  <div class="container">

    <!-- Add branding -->
    <div class="navbar-header">
      <!-- Style toggle button -->
      <button type="button"
              class="navbar-toggle"
              data-toggle="collapse"
              data-target="#myNavbar"
              aria-expanded="false">
        <span class="sr-only">Toggle navigation</span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
      </button>

      <a class="navbar-brand" href="">Website Name</a>
      <!-- Add a logo in a.navbar-brand. Height should 20px. -->
      <p class="navbar-text">optional: extra text</p>
    </div>

    <!-- .collapse is display: none; -->
    <div class="collapse navbar-collapse" id="myNavbar">
      <ul class="nav navbar-nav">
        <li><a href="#">Submenu 1</a></li>
        <li><a href="#">Submenu 2</a></li>

        <!-- bs:dropdown -->
        <li>
          <a href="#"
             class="dropdown-toggle"
             data-toggle="dropdown"
             role="button"
             aria-haspopup="true"
             aria-expanded="false">Submenu 3</a>
          <ul class="dropdown-menu">
            ...
          </ul>
        </li>

      </ul> <!-- ul.navbar-nav ends -->

      <!-- Add extra text -->
      <p class="navbar-text">optional: extra text</p>

      <!-- Add a bs:button -->
      <button type="button" class="btn btn-default navbar-btn">sign in</button>

      <!-- Add a bs:form or bs:form:input-group -->
      <form class="navbar-form navbar-right" role="search">
        <div class="form-group">
          ...
        </div>
      </form>

    </div>
    <!-- div.collapse ends -->

  </div>
  <!-- div.container ends -->
</nav>
    3. Breadcrumb

      Simple align links horizontally 

      <ol class="breadcrumb">
  <li class="breadcrumb-item"><a href="#">Home</a></li>
  <li class="breadcrumb-item"><a href="#">Library</a></li>
  <li class="breadcrumb-item active">Data</li>
</ol>
    4. Pagination and pager

      Simply align links horizontally BS3 recommends to replace <a> with <span> when li.disabled 

      <nav aria-label="Page navigation">
  <!-- .pagination-lg | .pagination-sm :: add to ul -->
  <ul class="pagination">
    <!-- li.disabled -->
    <li class="disabled">
  <a href="#" aria-label="Previous">
    <span aria-hidden="true">&laquo;</span>
  </a>
    </li>

    <!-- li.active -->
    <li class="active">
  <a href="#">1</a> <span class="sr-only">(current)</span>
    </li>

    <li><a href="#">2</a></li>
    <li><a href="#">3</a></li>
    <li><a href="#">4</a></li>
    <li><a href="#">5</a></li>

    <li>
  <a href="#" aria-label="Next">
    <span aria-hidden="true">&raquo;</span>
  </a>
    </li>
  </ul>
</nav>

      Pager :: Simple Previous and Next buttons with no pages 

      <!-- add <nav>, classes li.previous, li.next to align buttons to both ends -->
<!-- remove <nav>, classes li.previous, li.next to center them -->
<nav aria-label="...">
  <ul class="pager">
    <li class="previous disabled">
  <a href="#"><span aria-hidden="true">&larr;</span> Older</a>
    </li>
    <li class="next">
  <a href="#">Newer <span aria-hidden="true">&rarr;</span></a>
    </li>
  </ul>
</nav>

      BS4 structure is a little different

      .page-item
      add to every <li>
      .page-link
      add to every li>a
      .disabled
      add to li and add tabindex="-1" to li>a. <a> will have pointer-events: none.
      Optionally remove <a> if li.disabled
      BS3 seems to recommend this behavior
  13. Carousel (js)

    Don't add data-ride if it's triggered by javascript

    Pictures have to be the same width and height. 

    <!-- Options
data-interval :: 5000
data-pause :: "hover" | null
data-wrap :: true
keyboard :: true
-->
<div id="carousel-example-generic"
     class="carousel slide"
     data-ride="carousel">

  <ol class="carousel-indicators">
    <!-- An item must have .active -->
    <li data-target="#carousel-example-generic" data-slide-to="0" class="active"></li>
    <li data-target="#carousel-example-generic" data-slide-to="1"></li>
    <li data-target="#carousel-example-generic" data-slide-to="2"></li>
  </ol>

  <div class="carousel-inner" role="listbox">
    <!-- An item must have .active -->
    <!-- BS4 :: use .carousel-item instead of .item -->
    <div class="item active">
      <img src="..." alt="...">
      <div class="carousel-caption">
        <h3>...</h3>
        <p>...</p>
      </div>
    </div>

    <div class="item">
      <img src="..." alt="...">
      <div class="carousel-caption">
        ...
      </div>
    </div>

    <!--  more items -->
  </div>

  <!-- Controls -->
  <!-- data-slide could be
  prev, next, cycle, pause, number (integer starting from 0)
  -->
  <a class="left carousel-control"
     href="#carousel-example-generic"
     role="button"
     data-slide="prev">
    <span class="glyphicon glyphicon-chevron-left" aria-hidden="true"></span>
    <span class="sr-only">Previous</span>
  </a>
  <a class="right carousel-control"
     href="#carousel-example-generic"
     role="button"
     data-slide="next">
    <span class="glyphicon glyphicon-chevron-right" aria-hidden="true"></span>
    <span class="sr-only">Next</span>
  </a>

</div>

<script>
  $(function() {
    $('#carousel-example-generic').carousel({
      // options :: see above
    });
    /* Other methods refer to data-slide
    e.g. .carousel('cycle') ...
     */

    // Catch events
    // slide.bs.carousel :: start
    // slid.bs.carousel :: complete
    // Event object
    // direction :: 'left' | 'right'
    // relatedTarget :: DOM element that is about to be active
    $('#carousel-example-generic').on('slide.bs.carousel', function(e) {
     // do something
    });

  });
</script>
  14. Modal (js)
    <!-- When a modal is open, it adds .modal-open to <body> and also
  a .modalbackdrop to provide a click area to dismiss the modal
--->
<button type="button" class="btn btn-primary btn-lg"
    data-toggle="modal"
    data-target="#myModal">
  Launch demo modal
</button>

<!-- remove .fade to remove animation -->
<div class="modal fade"
     id="myModal"
     tabindex="-1"
     role="dialog"
     aria-labelledby="myModalLabel">

  <!-- Sizing :: in modal-dialog, add modal-lg, modal-sm -->
  <div class="modal-dialog" role="document">

    <div class="modal-content">

  <div class="modal-header">
    <button type="button"
        class="close"
        data-dismiss="modal"
        aria-label="Close">
      <span aria-hidden="true">&times;</span>
    </button>
    <h4 class="modal-title" id="myModalLabel">Modal title</h4>
  </div>

  <div class="modal-body">
    <!-- stack .row to use grid -->
    <!-- Any HTML or BS components -->
    <!-- form field autofocus has no effect. Refer to javascript below -->
  </div>

  <div class="modal-footer">
    <button type="button" class="btn btn-default"
        data-dismiss="modal">Close</button>
    <button type="button" class="btn btn-primary">Save changes</button>
  </div>

    </div>

  </div>

</div>

<script>
  /* Options :: data-*
     backdrop :: true. Use 'static' to not close the modal on click
     keyboard :: true. Esp key closes the modal
     show :: true. Show when initialized
     remote :: deprecated.
  */

  /* Methods
     .modal(options)
     method_name :: toggle, show, hide, and
     handUpdate (readjust positioning. Only needed if modal height changes while it opens)
     .modal('method_name')
  */

  /* Event types:
  show.bs.modal :: start.
  shown.bs.modal :: finished (after transition is complete)
  hide.bs.modal :: start to hide
  hidden.bs.modal :: finished being hidden
  loaded.bs.modal :: content has been loaded through remote option
     Event object
   e.relatedTarget :: If triggered by a click, it's the clicked element
   */
  $('#myModal').on('show.bs.modal', function(e) {
    // Using the same modal to display different content
    var button = $(e.relatedTarget);
    var data = button.data('lili-content');
    // Do some Ajax
    var modal = $(this);
    modal.find('.modal-title').text('...');
    modal.find('.modal-body').innerHTML = '...';

    <!-- form field autofocus has no effect. Use this workaround -->
    $('#fieldInputID').focus();
  });
</script>
  15. Collapse (js)

    bs:collapse :: Button to expand or collapse content Used in bs:navbar bs:panel bs:card (accordion) 

    <!-- Action Button as <a> -->
<a class="btn btn-primary"
   role="button"
   data-toggle="collapse"
   href="#collapseExample"
   aria-expanded="false"
   aria-controls="collapseExample">
  Link with href
</a>

<!-- Action Button as <button> -->
<button class="btn btn-primary"
        type="button"
        data-toggle="collapse"
        data-target="#collapseExample"
        aria-expanded="false"
        aria-controls="collapseExample">
  Button with data-target
</button>

<!-- .collapse :: hide content initially -->
<div class="collapse" id="collapseExample">
  <!-- BS4 :: use .card.card-block instead of .well -->
  <div class="well">
    ...
  </div>
</div>
  16. Tooltip (js)

    Need to initiate in javascript 

    <button type="button" class="btn btn-default" 
        data-toggle="tooltip" 
        data-placement="left" 
        title="Tooltip on left">Tooltip on left</button>

<a href="#"
   data-toggle="tooltip"
   data-placement="bottom"
   title="">Tooltip on bottom</a>

<script>
  $('[data-toggle="tooltip"]').tooltip();
</script>
  17. Popover (js)

    Need to initiate in javascript A title can be specified

    data-trigger can be click | hover | focus | manual 

    <a tabindex="0"
   class="btn btn-lg btn-danger"
   role="button"
   data-toggle="popover"
   data-container="body"
   data-trigger="focus"
   title="Dismissible popover"
   data-content="And here's some amazing content. It's very engaging. Right?">
  Click on anywhere to dismiss the popover
</a>

<button type="button"
   class="btn btn-lg btn-danger"
   role="button"
   data-toggle="popover"
   data-trigger="focus"
   title="Dismissible popover"
   data-content="And here's some amazing content. It's very engaging. Right?">
  Must click on the button again to dismiss the popover
</button>

<script>
  $('[data-toggle="popover"]').popover();
</script>
  18. Scrollspy (js)

    Must use with bs:nav 

    <body data-spy="scroll" data-target="#navbar-example">
  ...
  <div id="navbar-example">
    <ul class="nav nav-tabs" role="tablist">
      ...
    </ul>
  </div>
  ...
</body>

<style>
body {
  position: relative;
}
</style>
6.34.1.7. Mockup Tools

LayoutIt.com BS3 http://www.bootply.com/

6.34.1.8. v4
  1. Display utilities .d-* .display-*

    .d-{value} for xs .d-{breakpoint}-{value} for sm, md, lg, and xl.

    d-none d-inline d-inline-block d-block d-table d-table-cell d-table-row d-flex d-inline-flex

    Hidden on all .d-none Hidden only on xs .d-none .d-sm-block Hidden only on sm .d-sm-none .d-md-block Hidden only on md .d-md-none .d-lg-block Hidden only on lg .d-lg-none .d-xl-block Hidden only on xl .d-xl-none Visible on all .d-block Visible only on xs .d-block .d-sm-none Visible only on sm .d-none .d-sm-block .d-md-none Visible only on md .d-none .d-md-block .d-lg-none Visible only on lg .d-none .d-lg-block .d-xl-none Visible only on xl .d-none .d-xl-block

    Display headings <h1 class="display-1">Display 1</h1> <h1 class="display-2">Display 2</h1> <h1 class="display-3">Display 3</h1> <h1 class="display-4">Display 4</h1>

  2. p.lead
    .lead {
    font-size: 1.25rem;
    font-weight: 300;
}

<p class="lead">25% bigger than 1rem and 300 font weight.</p>

Can be used to enlarge a button
<p class="lead">
  <a class="btn btn-primary btn-lg" href="#" role="button">Learn more</a>
</p>
  3. Spacing

    mt, pt, mb, pb mx, px for left and right my, py for top and bottom

    {property}{sides}-{size} for xs {property}{sides}-{breakpoint}-{size} for sm, md, lg and xl

    size m-0 m-1 to $spacer * .25 m-2 $spacer * .5 m-3 $spacer m-4 $spacer * 1.5 m-5 $spacer * 3 m-auto

    $spacer is 1 rem

  4. flex

    https://getbootstrap.com/docs/4.0/utilities/flex/

    1. Use case

      Left and right 

      <section class="search-bar">
  <div class="page-wrap container">
    <div class="d-flex flex-wrap align-items-center">
      <h1 class="mr-auto p-2 text-white text-uppercase">New And Used Inventory</h1>
      <a href="/admin" class="p-2 btn btn-light btn-sm bg-white text-primary" role="button">Adminstrator Login</a>
    </div>
  </div>
</section>
    2. display
      .d-flex
.d-inline-flex
.d-sm-flex
.d-sm-inline-flex
.d-md-flex
.d-md-inline-flex
.d-lg-flex
.d-lg-inline-flex
.d-xl-flex
.d-xl-inline-flex
    3. flex-row
      .flex-row
.flex-row-reverse
.flex-column
.flex-column-reverse
.flex-sm-row
.flex-sm-row-reverse
.flex-sm-column
.flex-sm-column-reverse
.flex-md-row
.flex-md-row-reverse
.flex-md-column
.flex-md-column-reverse
.flex-lg-row
.flex-lg-row-reverse
.flex-lg-column
.flex-lg-column-reverse
.flex-xl-row
.flex-xl-row-reverse
.flex-xl-column
.flex-xl-column-reverse
    4. flex-wrap
      .flex-nowrap
.flex-wrap
.flex-wrap-reverse
.flex-sm-nowrap
.flex-sm-wrap
.flex-sm-wrap-reverse
.flex-md-nowrap
.flex-md-wrap
.flex-md-wrap-reverse
.flex-lg-nowrap
.flex-lg-wrap
.flex-lg-wrap-reverse
.flex-xl-nowrap
.flex-xl-wrap
.flex-xl-wrap-reverse
    5. justify-content
      .justify-content-start
.justify-content-end
.justify-content-center
.justify-content-between
.justify-content-around
.justify-content-sm-start
.justify-content-sm-end
.justify-content-sm-center
.justify-content-sm-between
.justify-content-sm-around
.justify-content-md-start
.justify-content-md-end
.justify-content-md-center
.justify-content-md-between
.justify-content-md-around
.justify-content-lg-start
.justify-content-lg-end
.justify-content-lg-center
.justify-content-lg-between
.justify-content-lg-around
.justify-content-xl-start
.justify-content-xl-end
.justify-content-xl-center
.justify-content-xl-between
.justify-content-xl-around
    6. align-items
      .align-items-start
.align-items-end
.align-items-center
.align-items-baseline
.align-items-stretch
.align-items-sm-start
.align-items-sm-end
.align-items-sm-center
.align-items-sm-baseline
.align-items-sm-stretch
.align-items-md-start
.align-items-md-end
.align-items-md-center
.align-items-md-baseline
.align-items-md-stretch
.align-items-lg-start
.align-items-lg-end
.align-items-lg-center
.align-items-lg-baseline
.align-items-lg-stretch
.align-items-xl-start
.align-items-xl-end
.align-items-xl-center
.align-items-xl-baseline
.align-items-xl-stretch
    7. auto margin
      <!-- all three left -->
<div class="d-flex">
  <div class="p-2">Flex item</div>
  <div class="p-2">Flex item</div>
  <div class="p-2">Flex item</div>
</div>

<!-- left right right-->
<div class="d-flex">
  <div class="mr-auto p-2">Flex item</div>
  <div class="p-2">Flex item</div>
  <div class="p-2">Flex item</div>
</div>

<!-- left left right -->
<div class="d-flex">
  <div class="p-2">Flex item</div>
  <div class="p-2">Flex item</div>
  <div class="ml-auto p-2">Flex item</div>
</div>
    8. iOS 8 and lower

      v4 doesn't support flex for iOS 8 and lower. 

      display: -webkit-box; /* v4 uses. and this doesn't support wrap so -webkit-flex has to be used */
display: -moz-box;
display: -ms-flexbox;
display: -webkit-flex; /* should use this */
display: flex;

/* v4 doesn't use any -webkit-flex-* */
-webkit-flex-wrap: wrap;
-webkit-justify-content: center;
        justify-content: center;
-webkit-align-items: center;
        align-items: center;

      Refer to modernizr:flexbox

  5. color

    https://getbootstrap.com/docs/4.0/utilities/colors/

    .bg-white .text-white

    primary :: blue secondary :: grey success :: green danger :: red warning :: yellow info :: cyan light :: light grey muted :: darker grey similary to secondary

  6. text

    .text-lowercase -uppercase -capitalize

  7. Popover
    Require popper.js
    npm install --save popper.js

    Need to be initialized using javascript 

    // initialize all
$(function () {
  $('[data-toggle="popover"]').popover()
})

//

    button not dismissable 

    <button type="button" 
 class="btn btn-lg btn-danger" 
 data-toggle="popover" 
 title="Popover title" 
 data-content="And here's some amazing content. It's very engaging. Right?">Click to toggle popover</button>

    <a> dismissable (click somewhere else) 

    <a tabindex="0" 
   class="btn btn-lg btn-danger" 
   role="button" 
   data-toggle="popover" 
   data-trigger="focus" 
   title="Dismissible popover" 
   data-content="And here's some amazing content. It's very engaging. Right?">Dismissible popover</a>

    Events :: *.bs.popover show, shown, hide, hidden, inserted 

    $('#myPopover').on('shown.bs.popover', function () {
  // do something…
})

    Methods

    trigger mode manual if you want to distinguish hover and click events 

    $('.action-call[data-toggle="popover"]').popover({trigger:'manual'})
    .on({
      mouseenter: function() {$(this).popover('show');},
      mouseleave: function() {$(this).popover('hide');},
      click: function() {window.open('tel:'+$(this).data('phone'));}
    });
6.34.1.9. Third party

https://mdbootstrap.com/

6.34.2. Tailwind

6.34.2.1. Install
  1. Via Tailwind CLI
    npm i -D tailwindcss
npx tailwindcss init

# Generated src/css/tailwind.config.js
  2. Via PostCSS
    npm install -D tailwindcss postcss autoprefixer
npx tailwindcss init

# Edit postcss.config.js

    import tailwind from 'tailwindcss'
import autoprefixer from 'autoprefixer'
import tailwindConfig from './src/css/tailwind.config.js'

export default {
  plugins: [tailwind(tailwindConfig), autoprefixer],
}
  3. Playground
6.34.2.2. Config
  1. Color
6.34.2.3. padding
p-1
0.25rem = 4px
p-2
0.5rem = 8px
p-4
1rem
6.34.2.4. About turning off preflight
6.34.2.5. Pseudo-class Reference
All
https://tailwindcss.com/docs/hover-focus-and-other-states#pseudo-class-reference
(no term)
hover:
(no term)
focus:
  • focus_within:
  • focus:visible
(no term)
active:
(no term)
visited:
(no term)
target:
  • URL #name open
(no term)
selection:
(no term)
<li class="marker:text-pink-500" list-disc>
marker:
the list marker (bullet/number)
list-disc
disk (dot) style
list-decimal
number
list-none
no bullet
6.34.2.6. Position modifiers
  • first:
    • first-of-type:
  • last:
    • last-of-type:
  • only:
    • only-of-type:
  • odd:
  • even:
  • empty:
    • empty:hidden
      • Hide if there's no content
6.34.2.7. Form modifiers
  • HTML state and corresponding Tailwind modifier
    disabled
    disabled:
    (no term)
    checked:
    indeterminate:
    group of radio buttons or checkboxes
    (no term)
    valid:
    • invalid
    (no term)
    in-range:
    • out-of-range:
  • file:
    • hover:file:
6.34.2.8. Child Modifiers and Sybling Modifiers
  • group:
    • group-hover:
  • peer:
    • peer-*:
      • peer-invalid:
6.34.2.9. Attribute modifier
  • <html dir="ltr">
    • ltr:
    • rtl:
  • <details class="group open:ring-2 open:p-3">
    • open:
    • group-open:
6.34.2.10. Media modifier
  • portait:
  • landscape:
  • print:
6.34.2.11. Dark mode
  • dark:
6.34.2.12. Arbitrary Values
<div class="xl:w-[80%]"></div>

<!-- Tailwind in some cases can auto resolve ambiguities -->

<!-- Will generate a font-size utility -->
<div class="text-[22px]">...</div>

<!-- Will generate a color utility -->
<div class="text-[#bada55]">...</div>

<!-- To resolve ambiguities manually, specify the CSS key or CSS data type -->

<p class="text-[color:var(--primary)] text-[clamp(1em,4vw,4em)]">Hello</p>
6.34.2.13. Tools built on top of Tailwind

6.34.3. MUI

  • https://www.figma.com/community/file/912837788133317724
  • Base UI e.g. @mui/base/Button is a library of unstyled React UI components and hooks
  • MUI System is a set of CSS utilities that provide a styled-components-like API
  • Material UI e.g. @mui/material/button is Base UI plus styles
  • MUI X is a collection of advanced UI components and should install with Material UI
6.34.3.1. MUI System
  • @mui/system
  1. styled
    • Add prop sx to non-MUI-component
    import { styled } from '@mui/material/styles';

const Div = styled('div')``;
  2. unstable_styleFunctionSx
    import * as React from 'react';
import styled, { ThemeProvider } from 'styled-components';
import { unstable_styleFunctionSx } from '@mui/system';
import { createTheme } from '@mui/material/styles';

const theme = createTheme();

const Div = styled('div')(unstable_styleFunctionSx);

export default function StyleFunctionSxDemo() {
  return (
    <ThemeProvider theme={theme}>
      <Div sx={{ m: 1, p: 1, border: 1 }}>Custom component with the sx prop</Div>
    </ThemeProvider>
  );
}

6.34.4. Pico.css

  • https://picocss.com/
  • Classless version or less amount of classes version
  • One CSS file, no JavaScript
  • Basic components
    • Modal
    • Accordion
    • Card
    • Dropdown
    • Nav
    • Progress
    • Loading
    • Tooltip

6.34.5. Animate.css

7. HTML

7.1. DOM

7.1.1. Window

pageYOffset
scrollY
same as dom:window:pageyoffset
innerHeight
window.innerHeight + window.pageYOffset is the y coordinate of the window

7.1.2. Document Object

Document properties and methods

document.methodName() or Notes Use in
document.propertyName   html:ElementObject
     
createElement("p") appendChild no
     
createTextNode("text node"); appendChild no
     
createAttribute('data-id') appendChild no
     
getElementById("id") null or html:ElementObject no
     
getElementsByClassName NodeList yes
     
getElementsByTagName("p") NodeList yes
     
querySelector('#id') 1st matched element. null or exception or html:ElementObject yes
     
querySelectorAll('#id') Static NodeList or exception yes
     
normalize() remove empty text nodes and join adjacent text nodes yes
     
readyState - loading :: the document is still loading no
html:document:readyState - interactive :: document has finished loading and  
  the document has been parsed but sub-resources e.g.  
  images, stylesheets and frames are still loading  
  - complete :: document and all sub-resources have finished  
  loading. The state indicates that the html:event:load  
  is about to fire  
designMode
default "off"

7.1.3. NodeList Object NodeList

NodeList Object appears like an array (loop like an array) but cannot use Array Methods such as valueOf(), join(), forEach().

Static NodeList means changes to DOM have no effect.

All NodeList objects are live objects meaning changes that are made in other code places reflect on all NodeList objects that were previously found.

var myNodeList = document.getElementsByTagName("P"); console.log(myNodeList.length); var firstP = myNodeList[0]; var firstP = myNodeList.item(0); / same as above, return null if not exists var firstP = document.getElementsByTagName("p")[0]; / same as above

console.log(firstP.innerHTML); firstP.style.backgroundColor = "red";

for (var i=0; i < myNodeList.length; i++) { myNodeList[i].style.backgroundColor = "red"; }

Other ways to loop NodeList 

Array.prototype.forEach.call(
    aNodeListArray,
    function(item) {
        console.log(item.id);
    }
);

[...myNodeList].forEach(
    function(item) {
        console.log(item.id);
    }
);

7.1.4. Element Object

7.1.4.1. Basics

Element Properties and Methods

element.methodName() or Notes Use in
element.propertyName   html:DocumentObject?
accessKey get/set accessKey  
addEventListerner() refer to event attributes without 'on' yes
  e.addEventListerner("click", myFunction,false )  
  false for bubbling, true for capturing  
removeEventListener() anonymous function will not work yes
     
click() simulate a click on an element no
blur()   no
focus()   no
     
innerHTML get/set no
     
appendChild(newChildNode) move element to the end of another element no
  appendChild  
removeChild(childNode)   no
list.insertBefore(newnode,    
list.childNodes[0])    
     
parentNode read only no
parentElement read only no
     
.cloneNode()    
     
attributes read only. NamedNodeMap no
getAttribute("class") attribute value in string no
getAttributeNode("class") Attr object no
     
.childElementCount the number of child element nodes not text  
  and comment nodes  
.childNodes NodeList object. .childNodes.length  
  var c = e.childNodes[0].text  
.children exclude text and comment nodes.  
  HTMLCollection object.  
classList read only. DOMTokenList no
className get/set no
     
clientHeight height + padding, no border and scrollbar no
clientWidth   no
clientTop Top border width no
clientLeft Left border width no
offsetWidth width+padding+border+scrollbar but no margin no
offsetHeight   no
.offsetLeft, offsetTop relative to offsetParent  
.scrollHeight, .scrollWidth width+padding only  
.scrollLeft, .scrollTop pixels are scrolled  
     
e1.compareDocumentPosition(e2) compare positions of elements  
e1.contains(e2) if e2 is a descendant of e1  
.contentEditable    
     
.firstChild child node  
.firstElementChild child element node  
.nextSibling    
.nextElementSibling    
     
.hasAttribute("onclick")    
.hasAttributes() has any attributes?  
.hasChildNodes() has any child nodes  
     
.isEqualNode(e1,e2)    
     
.nodeName tag name  
.tagName tag name in upppercase  
.nodeValue    
     
7.1.4.2. Reflow
  • Reflow is a user-blocking operation that computes the layout of the document. A reflow on an element is the calculation of its dimensions and position in the document
7.1.4.3. elem.offsetWidth, elem.clientWidth, elem.scrollWidth, getComputedStyle(elem).width, elem.getBoundingClientRect().width
  • offsetWidth, offsetHeight
    • width + padding + border + scrollbar, but no margin
    • Rounded to integer in pixels
    • it is faster than clientWidth
  • clientWidth, clientHeight
    • height + padding, but no margin, no border, no scrollbar
    • 0 for inline elements and elements with no CSS
    • rounded to integer in pixels
  • scrollWidth
    width + padding, but no margin, no border, no scrollbar
    similar to clientWidth
    (no term)
    rounded to integer in pixels
  • getComputedStyle(elem).width
  • elem.getBoundingClientRect().width
    • https://developer.mozilla.org/en-US/docs/Web/API/Element/getBoundingClientRect

    • Returns a DOMRect object which has 4 keys that hold fractional values

      • size of an element (width, height) and its position relative to the viewport x eq. left, y eq. top, right, bottom
      • Returned size width is very close to offsetWidth but different
        • width or height + padding + border-width,
        • width or height only if box-sizing: border-box
        • If the element is transformed e.g. width: 100px; transform: scale(0.5);, it will return 50 as the width
    • Also works for virtual DOM e.g. Vue this.$refs['some-ref'].getBoundingClientRect().height

7.1.5. Event Object

7.1.5.1. Basics
<input id="elem" type="button" value="Click me">
// event is the first argument
<script>
  var elem = document.getElementById("elem");
  elem.onclick = function(e) {
    alert('Thank you');
  };

  elem.setAttribute('onclick', "javascript:console.log('hello')");

  elem.addEventListener('click', function (e) {
   // e is SomeEvent object
  });

</script>

// Pass event for inlince onclick
<pre onclick="doSth(event)">

function doSth(e) {
  e.stopPropagation();
}

<pre onclick="event.stopPropagation();doSth();">

Property

type event name. e.g. mousedown
timeStamp event occured
bubbles true or false
defaultPrevented if method preventDefault() was called
currentTarget the element event happens on
target the (child) element event happens on
view get reference of the Window object where the event occured
   

Method

preventDefault() stop defeault event action
stopImmediatePropagation() stop other listeners of the same event (e.g. click) on the same target
stopPropagation() stop bubbling
7.1.5.2. Dispatch event, trigger event
<input id="lili" type="text" onchange="console.log('value is changed. New value' + this.value, event);">

<button onclick="document.getElementById('lili').value=123;document.getElementById('lili').onchange();">Change me, event object is not passed onto #lili.onchange</button>

<button onclick="document.getElementById('lili').value=321;document.getElementById('lili').dispatchEvent(new Event('change'));">Change me, event object is passed onto #lili.onchange, just like a normal UI operation</button>
7.1.5.3. onclick
  • Consider assign the same event handler for keydown event for devices such as TV's which don't use mouse nor a touch screen
  • Returning false means the click does not go through
7.1.5.4. onchange
  • onchange actually happens after oninput and then after onblur
  • iOS doesn't trigger onchange if the input value is changed programmatically

  • In Safari, if onblur modifies the input value programmatically, and the value is different from the last oninput, then onchange will not fire! 

    // onblur event handler
// may change text.value based on some conditions
var isSafari = /^((?!chrome|android).)*safari/i.test(navigator.userAgent);
if (focus && text.value !== oldAmount && text.onchange && isSafari) {
text.dispatchEvent(new Event('change'));
}
7.1.5.5. Events about checkbox
<div class="inputrow">
    <input type="checkbox" onchange="" onclick="console.log(this.checked);" id="aCheckbox" value="1" checked>
    <label for="aCheckbox">Checkbox Label</label>
</div>
  • Initially, the checkbox is checked
  • If the label or the checkbox is clicked, onclick is triggered and logs this.checked is false
    • return false in onclick will stop saving this.checked = false, this.checked will remain as true
      • and onchange is not run
7.1.5.6. Events about radio
<input type="radio" name="my_radio" id="my_radio_0" onclick="doSomething()" value="0">
<label for="my_radio_0">None</label>
7.1.5.7. MouseEvent Object
altKey whether ALT key was pressed
shiftKey whether Shift key was pressed
ctrlKey whether Ctrl key was pressed
button 0 left click, 1 middle, 2 right
which button + 1: 0: no button, 1 left, 2 middle, 3 right
buttons one or more buttons. More button such as browse back button.Safari not supported
clientX relative to current window (viewport).
clientY  
pageX relative to the fully rendered content area. Height might be larger than
pageY viewport height.
screenX relative to the screen. e.g. number of monitors and monitor screen
screenY  
relatedTarget use with mouseover event to find the element the cursor jsut exited or
  with the mouseout event to find the element the cursor just entered
  • onclick on <a href="#"> a tag
    onclick="doSomething();return false"
    eq. to applying both event.preventDefault(); event.stopPropagation();
    (no term)

    onclick="doSomething(event);" 

    function doSomething(e, myVal){
//doing custom things with myVal

//here I want to prevent default
e = e || window.event;
e.preventDefault();
// e.stopPropagation();
}
7.1.5.8. KeyboardEvent Object
Keyboard code (KC)
actual key on keyboard (always lowercase)
Unicode Character code (UCC)
result Unicode character (Shift + w = W, return the character code of W)
altKey whether alt is pressed
ctrlKey  
shiftKey  
metaKey whether meta is pressed. Windows key
key key name on keyboard (F1,Enter..)
keyCode keypress-0, keydown,keyup-UCC. All browsers.
charCode keypress-UCC, keydown,keyup-0. All browsers.
which keypress-UCC, keydown,keyup-UCC. All browsers.
location 4 numbers 0 - standard,1 - left,
  2 - right (CTRL), 3 - numpad
   
  • If you want to return UCC for all events cross all browsers use this var x = event.charCode || event.keyCode;
7.1.5.9. HashChangeEvent Object

newURL :: the URL after the hash has been changed oldURL

7.1.5.10. PageTransitionEvent

For events: onpageshow and onpagehide persisted :: whether the page was cached

7.1.5.11. FocusEvent Object

relatedTarget :: the element related to the element that triggered the event event.relatedTarget.tagName;

7.1.5.12. AnimationEvent Object

animationName :: keyframe name elapsedTime :: number of seconds the transition has been running

7.1.5.13. HTML DOM Event
  1. DOMContentLoaded html:event:DOMContentLoaded
    Target can be document and window
    document.addEventListener('DOMContentLoaded', (e) => { ... });
    (no term)
    It's fired when the initial HTML has been completely loaded and parsed, without waiting for stylesheets, images and subframes to finish loading. html:event:load is fully-loaded
    (no term)
    Synchronous JavaScript can issue a doc.write at any point; hence the DOM tree construction is blocked anytime a synchronous script is encountered
    (no term)
    JavaScript can query for a computed style of any object, which means it can also block on CSS
    (no term)
    DOM construction can't proceed until JavaScript is executed, and JavaScript can’t proceed until CSSOM is available
    (no term)
    Mark JavaScript files as js:defer to unblock construction of DOM and execute them before domContentLoadedEventStart but JavaScript files still need to wait for CSSOM to be finished
    (no term)
    Mark JavaScript files as js:async is similar to js:defer but DCL does not have to wait for execution of async scripts
    (no term)
    By default, JavaScript will block DOM construction, which may block on CSSOM. Sync scripts are bad, but you already knew that. Marking scripts with “defer” and “async” makes an implicit promise to the document parser that you will not use doc.write, which in turn allows it to unblock DOM construction
    (no term)
    If at any point we must wait for JavaScript execution, then we will have to first wait for the CSSOM construction to finish. In other words, there is a hard dependency edge between JavaScript and CSS… Stylesheets at the top, scripts at the bottom? Now you know why
    (no term)
    https://calendar.perfplanet.com/2012/deciphering-the-critical-rendering-path/
  2. load html:event:load
    Target is window
    window.addEventListener('load', (e) => {})
    (no term)
    Refer to html:event:DOMContentLoaded

    Detailed events using PerformanceNavigationTiming

    domLoading
    browser is about to start parsing the first received bytes of the HTML document
    domInteractive
    browser has finished parsing all HTML and DOM construction
    domContentLoaded
    Both DOM is ready and there're no stylesheets that are blocking JavaScript execution (CSSOM is parsed and ready). Ready to construct the render tree.
    • domContentLoadedEventStart
    • domContentLoadedEventEnd
    domComplete
    all processing is complete and all resources e.g. images have finished downloading. The loading spinner has stopped spinning.
    load
    page is fully loaded
    • loadEventStart
    • loadEventEnd
7.1.5.14. readystatechange html:event:readystatechange
// Alternative to load event
document.onreadystatechange = function () {
  if (document.readyState === 'complete') {
    initApplication();
  }
}

document.addEventListener('readystatechange', event => {
  if (event.target.readyState === 'interactive') {
    initLoader();
  }
  else if (event.target.readyState === 'complete') {
    initApp();
  }
});
7.1.5.15. MessageEvent

7.1.6. DOMTokenList Object

A collection of space-separated tokens.

DOMTokenList.methodName() Notes
or  
DOMTokenList.propertyName  
length read only
add("class2Name")  
remove("class2Name")  
replace("old","new")  
toggle('class2Name') if exists, removes and return false. If not, add and return true
contains("class2Name")  

7.1.7. NamedNodeMap Object

A collection of Attr object 's. Loopable like an Array but no Array methods.

7.1.8. Attr Object

An attribute node.

name  
value  

7.2. Global Attributes

Can be applied to any element
https://developer.mozilla.org/en-US/docs/Web/HTML/Global_attributes
(no term)
id class
accesskey
Add shortcut. M-accesskey
download
filename. force download
contenteditable
true or false. Become editable
contextmenu
right click menu. Only in Firefox
dir
ltr rtl or auto. text direction
hidden
spellcheck
true or false
title
tool tip

7.2.2. Drag-and-Drop

  • ondrop ondragover draggable ondragstart 

    <!--
 ondropover allows this tag to receive dropped elements.
 ondrop defines what to do with the traferred data
-->
<div id="div1" ondrop="drop(e)" ondragover="allowDrop(e)"></div>

<!--
 Make a tag draggable: draggable="true"
 <a> and <img> are by default draggable
 ondragstart defines what data to pass to ondrop
-->
<img id="drag1" src="http://placehold.it/300x250" draggable="true" ondragstart="drag(e)">

<script>
function drag(e) {
  // Pass the dragged element id
  ev.dataTransfer.setData("text", ev.target.id);
}

function allowDrop(e) {
  e.preventDefault();
} 

function drop(e) {
  e.preventDefault();
  var data = e.dataTransfer.getData("text");
  e.target.appendChild(document.getElementById(data));
}
</script>

7.2.3. tabindex

  • Elements by default are focusable
    • a with href
      • Better having every a to have href attribute as even though adding tabindex attribute with a positive integer enables focus but cannot enter or spacebar to trigger onclick
    • link with href
    • button
    • input that are not type="hidden"
    • select
    • textarea
  • MacOS
    • By default, MacOS only tabs through Text Boxes and list only, System Preferences > Keyboard Preferences
      • lists, text fields and textareas. No buttons and checkboxes
  • Bringing an element to focus
    • Hit tab on the keyboard
    • Use JavaScript e.g. document.getElementById('a').focus()
  • integer
    -1 or any negative number
    make it unfocusable
    0
    put it in the default focus order determined by its position in the HTML
    (no term)
    positive number
    • Elements with positive tabindex will be placed in front of elements that don't have a tabindex attribute
    • The smaller, the higher order starting from 1
    • To test which is the first tab, click on the URL bar and then tab

7.3. Events and Event Attributes

  • All DOM Events
  • an event handler can have one of the 2 mechanisms:
    Capturing phase
    DOM heirarchy up
    • Run outer-most ancestor's (<html>) same event handler e.g. onclick
    • Then its child's same event handler, go on until the original element
    Bubbling phase (default)
    DOM heirarchy down
    • Run the original element's same event handler
    • Then its parent's same event handler, go on until the outer-most ancestor
    (no term)
    If an element has multiple event handlers of both mechanisms, the capturing phase will run first followed by bubbling
    (no term)
    There's another phase called target phase
    • Which is activated for event handlers that are registered using element.onclick or using HTML attribute. These event handlers are only triggered on the original elements, will not capture or bubble
  • These DOM or JavaScript objects have implemented EventTarget interface which can receive events and may have listeners
  • Add an event

    • element.onsomeeventlowercase 

      var btn = document.getElementById('button-a');
btn.onclick = function() {} // anonymous function

function bgCharge() {}
btn.onclick = bgCharge; // named function

// cannot have multiple event handlers
    • addEventListner() and removeEventListener()

    • https://developer.mozilla.org/en-US/docs/Web/API/EventTarget/addEventListener 

      btn.addEventListener('click', function() {});
btn.addEventListener('click', bgCharge);
btn.removeEventListener('click', bgCharge);

// only one of the identical event handlers can be registered
// anonymous functions, even though they have the same code, they are considered different

// target.addEventListener(type, listener [, options]); // options is an object
// target.addEventListener(type, listener [, useCapture]);
// useCapture default is false to use bubbling phase, true to capturing phase

    • Event delegation. e.g. in bubbling phase, add an event handler to the parent element of child elements, and this event handler knows which child element is clicked. This avoids adding unknown amount of event handlers to new/removed child elements 

      document.getElementById("myULelements").addEventListener("click",function(e) {
// e.target was the clicked element
if (e.target && e.target.matches("li.classA")) {
    console.log("Anchor element clicked!");
}
});
  • Event objects

7.3.1. Windows Event in body tag

onafterprint, onbeforeprint only IE and FF
onbeforeunload before page is closed/refreshed
onunload after page is closed/refreshed
onhashchange when URL hash is changed
onload when an object is loaded:
  body, frame, frameset, iframe,
  img, link, script, style
  Not run when it's loaded from cache
onpageshow similar to onload but always triggers
onpagehide similar to onunload but onunload event causes
  the page to not be cached.
onresize  
   

7.3.2. Form Event Attributes

Also can be applied on almost all elements.

oncontextmenu right click to bring context menu
onreset  
onsearch <input type="search" onsearch="">. IE and FF not supported
onselect when text is selected in <input type="text"> or <textarea>
onblur leave a form field. opposite of onfocus
onfocus opposite of onblur
onfocusin similar to onfocus but bubbles. FF not supported
onfocusout similar to onblur but bubbles.FF not supported
onchange oninput and lose focus. also works on <select> and <keygen>
oninput not working on <select> and <keygen>
   
oninput
<input>, <select>, <textarea>. Also works for contenteditable document.designMode

7.3.3. Mouse Events, Clipboard Events, Keyboard Events

ondblclick double click
   
ondragstart  
ondrag being dragged
ondragend  
   
ondragenter when dragged element enters the drop target
ondragover over the drop target
ondragleave leave the drop target
ondrop is dropped on the drop target
   
onscroll  
onwheel only Chrome and FF
   
oncopy  
oncut  
onpaste  
   
onmousedown  
onmouseenter when move into child element
onmouseleave when moving out of the element (not its children)
onmousemove when moving inside an element
onmouseover when onto this element
onmouseout when moving out of the element and its children
   
onkeydown evens are in order. works for all keys
onkeypress Some keys are not fired: C, A, S, ESC, etc.
onkeyup when releasing a key

7.3.4. Media Events

Elements: audio, img, embed, object, video Events in order:

onloadstart  
ondurationchange  
onloadedmetadata  
onloadeddata  
onprogress  
oncanplay  
oncanplaytrhough  
   

7.3.5. Animation Events

Refer to CSS @keyframes. animationstart animationend animationiteration :: when animaiton repeats

7.3.6. Transition Event

transitionend

7.3.7. Misc events

onerror when an error occurs while loading an external file: <img>

7.4. Canvas

7.5. Tags

7.5.1. Tags

Tag Usage Notes
a download="filename" force download
abbr title="World Health Organization" abbreviation WHO
address   author or owner (not just address) of a document
    or an article. display:block, italic author
base href,target base url and target for <a> tags
     
article   affect outline
aside    
section headline tag is required affect outline
nav   affect outline
     
cite <cite>U of T Professor</cite> wrap a person's title
dfn <dfn>HTML</dfn> is … A term to be defined
mark highlight text background yellow and text in black
s   text-decoration: line-through
q <q cite="abc.com">Build</q> Double quotes
ruby <ruby>a <rt>sound</rt></ruby>  
wbr <wbr>alongword</wbr> the word will not be broken for line break
     
meter min,max,high,low,optimum value
progress max, value  
     
     
colgrup, specify styles and span for  
col columns in a table  
     
iframe srcdoc="" IE & Edge not supported. Repalce double quote with &quot;
  sandbox="" empty to apply all restrictions
    e.g. allow-forms allow-same-origin
input autocomplete="on or off" default on. turn off for a specific field
  autofocus=""  
  formmethod="get or post" overwrite <form> type="submit, image"
  formaction="another.php" overwrite <form> type="submit, image"
  formenctype="multipart/form-data" overwrite <form> type="submit, image"
  formtarget="_blank" overwrite <form> type="submit, image"
  min="1" max="1979-12-31" for type="date" or type="number"
  multiple for type="file"
  pattern="regex" for type=text,date,search,url,tel,email,password
    escape " in regex with \x22
  placeholder="hint"  
  step="3" -3,0,3,6 can be accepted. type="number"
  type=email, url  
  required no in IE
  type="range" min max slide control
  type="search"  
  disabled="disabled" not selectable, editable and not submitted
  readonly="readonly" selectable, not editable and get submitted
samp sample computer output  
code monospace  
kbd monospace  
var variable name  
     
li value="100" increment by 1
     
  • Some HTML5 tags
    • <main>
      • Use it once and it is descendant of other tags
    • <header>
    • <nav>
    • <section>
    • <aside>
    • <article>
    • <footer>

7.5.2. <a>

7.5.2.1. rel attribute
  • Attribute value is called a keyword. Keywords are
    • noreferrer
      • For <a>, <area>, <form>
      • Instruct the browser to omit the Referrer header when navigating to the target resource
      • Additionally, behave as if another keyword 1.2 is used
    • noopener
      • For <a>, <area>, <form>
      • Instruct the browser to not granting the new browsing context access when navigating to the target resource
        • Window.opener property on the opened window is null

7.5.3. <iframe> attribute sandbox

  • Add space-separated allow-* tokens as value in sandbox attribute to lift particular restrictions. By default, all are not allowed
    allow-forms
    form submission
    allow-scripts

7.5.4. <picture>, <img srcset sizes>

7.5.4.1. html:img:srcset html:img:sizes
  • IE 11 doesn't support srcset nor sizes
  • it's used to serve larger—but otherwise identical—image sources to high resolution displays only
    When x is used in srcset, sizes cannot be set
    <img srcset="examples/images/image-384.jpg 1x, examples/images/image-768.jpg 2x" alt="…">
  • w in srcset is usually the actual image's width
  • refer to wp:img:srcset:sizes
  • sizes attribute is the width that the image needs to have under matched the media condition
    • If sizes has media condition, only the first matched media condition from the left is chosen
    • If the image needs to take a third of the viewport, then sizes="33.3vw"
    • Other than px, vw, em, rem can also be used in sizes. But not percentage
    • The srcset which has the closest w that matches the chosen sizes will serve:
      • First choose srcset that have w which is larger than sizes, among those choose the smallest srcset
      • If no srcset has w that is larger than sizes, choose the closest one (biggest srcset)
      • e.g. 3 srcset: 100w, 200w, 300w and the chosen sizes is 150px, then 200w is chosen. If sizes is 400px, then 300w is chosen

In order to test img with srcset and sizes, you need to open InCognito and resize the window size not choosing a simulated device. 

<img
  src="examples/images/fallback-smallest-size.jpg"
  sizes="(min-width: 40em) 80vw, 100vw"
  srcset="examples/images/medium.jpg 375w,
      examples/images/large.jpg 480w,
      examples/images/extralarge.jpg 768w"
  alt="…">

<style>  
  /* make sure img tag is responsive */
  img {
  max-width:100%;
  height:auto;
  }
</style>
7.5.4.2. html:picture

Used when you need explicit control over which source is shown at set viewport sizes. Except IE 11 and below 

<picture>
 <source srcset="a.jpg" media="(min-width: 600px)">
 <source srcset="b.jpg" media="(min-width: 500px)"> 
 <source srcset="fallback.jpg">
 <img src="fallback.jpg" alt="one alt">
</picture>

<!-- This is called art direction -->
<picture>
  <source media="(min-width: 800px)"
  sizes="100vw"
  srcset="cropped-for-wide-screens--large.jpg 1600w,
      cropped-for-wide-screens--small.jpg 800w" />
  <source media="(min-width: 600px)"
  sizes="100vw"
  srcset="full-image-for-standard-screens--large.jpg 1200w,
      full-image-for-standard-screens--small.jpg  600w" />
  <img
    src="zoomed-in-for-small-screens--small.jpg"
    srcset="zoomed-in-for-small-screens--large.jpg 800w,
        zoomed-in-for-small-screens--small.jpg 400w" alt="" />
</picture>

<picture>
    <source srcset='paul_irish.jxr' type='image/vnd.ms-photo'>
    <source srcset='paul_irish.jp2' type='image/jp2'>
    <source srcset='paul_irish.webp' type='image/webp'>
    <img src='paul_irish.jpg' alt='paul'>
</picture>
7.5.4.3. srcset and sizes calculation

small is 500x100, medium is 1000x200, and large is 2000x400 Device with screen width of 320px and 1x display (non-retina) Then chooses the one that is closest to 1 500 / 320 = 1.5625 1000 / 320 = 3.125 2000 / 320 = 6.25 

<img src="small.jpg" srcset="medium.jpg 1000w, large.jpg 2000w">

By default sizes="100vw", which is 100% viewport (320px in the above example) Extra media query can be added 

sizes="(max-width: 300px) 100vw, 300px"

Which means if media query matches, use 100vw. If not match, use 300px

Responsive image which takes 100% width. Define the largest width image first This is the WordPress way. wp:img:srcset:sizes 

<img src="medium.jpg" width="1000" height="200" 
     srcset="large.jpg 2000w, small.jpg 500w"
     sizes="(max-width: 1000px) 100vw, 1000px">
<style>
  /* make sure img tag is responsive */
  img {
  max-width:100%;
  height:auto;
  }
</style>

<picture>
  <source media="(max-width: 799px)" srcset="elva-480w-close-portrait.jpg">
    <source media="(min-width: 800px)" srcset="elva-800w.jpg">
  <img src="elva-800w.jpg" alt="Chris standing up holding his daughter Elva">
</picture>

CSS fallback 

<img srcset="
         examples/images/image-384.jpg 1x, 
         examples/images/image-768.jpg 2x
         " alt="…">

<style>
  .img {
  background-image: url(examples/images/image-384.jpg); 
  }
  @media 
  (-webkit-min-device-pixel-ratio: 2), 
  (min-resolution: 192dpi) { 
  .img {
  background-image: url(examples/images/image-768.jpg);
  }
  }

  /* 1.25 dpr */
  @media 
  (-webkit-min-device-pixel-ratio: 1.25), 
  (min-resolution: 120dpi){ 
  /* Retina-specific stuff here */
  }

  /* 1.3 dpr */
  @media 
  (-webkit-min-device-pixel-ratio: 1.3), 
  (min-resolution: 124.8dpi){ 
  /* Retina-specific stuff here */
  }

  /* 1.5 dpr */
  @media 
  (-webkit-min-device-pixel-ratio: 1.5), 
  (min-resolution: 144dpi){ 
  /* Retina-specific stuff here */
  }
</style>

7.5.5. <dl>, <dt>, <dd> Definition Term/name in a Definition list

<dl>
  <dt>Term #1</dt>
    <dd>Description of Term #1</dd>
  <dt>Term #2</dt>
    <dd>Description 1 of Term #2</dd>
    <dd>Description 2 of Term #2</dd>
</dl>

7.5.6. <blockquote> and <q>

<!-- blockquote and cite -->
<blockquote>
  <p>...</p>
  <cite>- Jeremy Keith</cite>
</blockquote>

<!-- inline quote, it has different styles based on different language -->
<p lang="fr">
  Jeremy Keith said,
  <q>...</q>
</p>

7.5.7. <time> 

<time>May 8</time>
<time datetime="2025-05-08">Tomorrow or any string</time>
<time datetime="07:00">7:00 am</time>
<time datetime="07:00:28.5">hh-mm-ss.ddd</time>
<time datetime="15:45-05:00">hh-mm-ss.ddd[+-]hh:mm With timezone</time>
<time datetime="2020-11-04T19:00">Wednesday, Nov 4th at 7pm</time>
<time datetime="2020-11-04 19:00-0500">Wednesday, Nov 4th at 7pm</time>

7.5.8. <figure> <figcaption>

<figure>
  <img src="a.png" width="960" height="720" alt="shiny black dog in the sun">

  <figcaption>Maggie the dog enjoys resting in a field, after a long day of chasing squirrels.</figcaption>
</figure>

7.5.9. <video> html:video

Codec
a device or program that encodes or decodes a data stream or signal
  • H.264
  • H.263
  • QuickTime Movie
(no term)
MP4 is the most compatible video format. Uses H.264 (not free, not open source) video codec and AAC audio codec
(no term)
WebM is Google for HTML5. Uses VP8 video codec and Vorbis audio codec. Doesn't support in iOS, Safari nor old IE
(no term)
FLV is Flash only. Uses H.263 video codec and MP3 audio codec
7.5.9.1. autoplay attribute

  • It's turned off on iOS mobile < v10 

    <video autoplay muted loop id="myVideo">
  <source src="rain.mp4" type="video/mp4">
</video>
7.5.9.2. preload attribute html:video:preload
  • <video id="video" preload="metadata" src="file.mp4" controls></video>
  • Instead of using <video preload="">, we can use html:link:rel:preload

    • Preload full small media files < 5mb 

        <link rel="preload" as="video" href="https://cdn.com/small-file.mp4">

  <video id="video" controls></video>

  <script>
// Later on, after some condition has been met, set video source to the
// preloaded video URL.
video.src = 'https://cdn.com/small-file.mp4';
video.play().then(_ => {
  // If preloaded video URL was already cached, playback started immediately.
});
  </script>

    • Preload the first segment use Media Source Extensions JavaScript API 

        <link rel="preload" as="fetch" href="https://cdn.com/file_1.webm">

  <video id="video" controls></video>

  <script>
const mediaSource = new MediaSource();
video.src = URL.createObjectURL(mediaSource);
mediaSource.addEventListener('sourceopen', sourceOpen, { once: true });

function sourceOpen() {
  URL.revokeObjectURL(video.src);
  const sourceBuffer = mediaSource.addSourceBuffer('video/webm; codecs="vp09.00.10.08"');

  // If video is preloaded already, fetch will return immediately a response
  // from the browser cache (memory cache). Otherwise, it will perform a
  // regular network fetch.
  fetch('https://cdn.com/file_1.webm')
  .then(response => response.arrayBuffer())
  .then(data => {
    // Append the data into the new sourceBuffer.
    sourceBuffer.appendBuffer(data);
    // TODO: Fetch file_2.webm when user starts playing video.
  })
  .catch(error => {
    // TODO: Show "Video is not available" message to user.
  });
}
  </script>
7.5.9.3. Caption
webvtt
web video text tracks. Plain text with .vtt extension
(no term)
kind attribute
descriptions
for blind people
chapters
allow video player to jump to a certain chapter
<video controls>
  <source src="moonwalk.480p.vp9.webm" type="video/webm">
  <source src="moonwalk.480p.h264.mp4" type="video/mp4">

  <track src="moonwalk.vtt"
     kind="captions"
     label="English"
     srclang="en"
     default>

   <track src="moonwalk.es-la.es.vtt"
     kind="subtitles"
     label="Español"
     srclang="es">

  <p>This would be a video of a moonwalk, if your device supported playing this video.</p>
</video>

7.5.10. <audio> 

<audio controls loop autoplay>
    <source src="b.ogg"
        type="audio/ogg; codec=opus">
    <source src="a.mp3"
        type="audio/mpeg">
    Sorry your browser doesn't not support audio.
</audio>

7.5.11. <tbody>, <thead> and <tfoot>

  • All 3 are to group multiple rows <tr>'s
    • To scroll <tbody> independently of <thead> and <tfoot>
  • <tbody>
    • Parent is <table>
    • Used to group rows <tr>. Multiple groups are allowed
    • Use a <tr> filled with <th>'s to create headers within each <tbody>
      • These <th>'s have different browser native CSS styles
  • <thead> and <tfoot>
    • Parent is <table>
    • Only one is permitted in a <table>

7.5.12. <input>

7.5.12.1. type=number
  • When attribute step is supported, UI widget (arrow up/down keys and validation) is available
    Not supported on mobile
    iOS Safari, Android browser, Chrome for Andriod
    Supported on desktop browsers
    for IE, Edge, FF, Chrome, Safari, Opera
  • Attributes step, min, max are part of the validation and increment/decrement
    • value is greater than min and less than max
  • UI widget happens only when a form is submitted. Validation cannot be triggered using JavaScript..
<input type="number">
<input type="number" step="0.01" class="currency">
<input type="number" step="0.01" min="0" class="currency-positive">
<input type="number" step="any" class="any-decimal">
7.5.12.2. type=range
  • Supported on almost all browsers
  • min, max, step
7.5.12.3. type=email
  • Only supported on modern mobile browsers
  • Does not validate email at all. Use this (val.replace(/\S+@\S+\.\S+/g, 'x') !== 'x' || val.value === 'x') ? 0 : 1

7.5.13. <table> 

<table>
  <tr>
    <td style="white-space: nowrap;">
  Force cell text not to be wrapped.
    </td>
  </tr>
</table>

7.6. Meta

7.6.1. Meta Refresh (redirect)

<meta http-equiv="refresh" content="0;URL='http://abc.com'" />
<!-- "0" means zero second -->

7.6.2. Open Graph Protocol html:og

7.6.3. Twitter Cards twitter:card

7.6.4. Address Bar html:address bar

Hide Safari UI (URL address bar) 

<meta name="apple-mobile-web-app-capable" content="yes">

<!--
By default, iOS web content displays under the top black status bar. 
Push the content to the top and make the status transparent
Default is black
-->
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">

<!-- Address Bar Color -->
<meta name="theme-color" content="#4285f4">

7.6.5. Meta viewport

  • Without this tag, the browser viewport will take the full resolution (window.screen.width * window.screen.devicePixelRatio)
  • Not wise to use min-device-width and max-device-width in @media CSS
<meta name="viewport" content="width=device-width, minimum-scale=1, initial-scale=1, shrink-to-fit=no">
<!--
Set the width of the page (viewport) to follow the screen-width of the device (screen-width can be changed by resizing the browsing or changing the orientation)
Initial zoom level 1.0
shrink-to-fit is Safari only

Prevent zoom: add maximum-scale=1.0, user-scalable=0
the above won't be WCAG compliant
-->

7.6.6. <meta charset="utf-8">

7.6.7. Response header html:meta:http-equiv:response header

HTTP response header can be defined in html markup 

<meta http-equiv="Content-Security-Policy" content="default-src https://cdn.example.net; child-src 'none'; object-src 'none'">

7.7. Header

All headers
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
Response header in html meta
html:meta:http-equiv:response header

7.7.1. Age header:age

A non-negative integer of seconds that the object (response) has been in a proxy cache (CDN, Varnish)

7.7.2. Cache-Control, Pragma - response header:cache-control

  • https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control
  • Refer to apache:mod_headers:cache
  • Pragma is for legacy HTTP/1.0 where Cache-Control HTTP/1.1 is not yet present. Only Pragma:no-cache or Pragma:cache
  • Directives
    must-revalidate
    force to verify the status before using the cache and expired ones should not be used.
    no-cache
    browser has to always check Etag and makes a roundtrip request. If file in server is new, it will be downloaded.
    no-store
    no cache should be stored in request or response e.g. Go Back to previous page, the browser shows the browser cache version of the page
    no-transform
    No transformations or conversions should be made to the resource. The Content-Encoding, Content-Range, Content-Type headers must not be modified by a proxy. A non-transparent proxy might, for example, convert between image formats in order to save cache space or to reduce the amount of traffic on a slow link. The no-transform directive disallows this.
    public
    response can be cached in all devices. It's not necessary as max-age inidicates the response is cacheable
    private
    response can be cached for a single user (user's browser) but not in shared cache (public CDN, Varnish or other intermediaries)
    proxy-revalidate
    max-age=<seconds>
    refer to header:expires If max-age is set in Cache-Control and Etag is set, then after max-age, client sends a request with If-None-Match: <etag-value> to server and if the Etag response doesn't change, server will return 304 Not Modified response and client will not download the file. The cache serves the same file for another max-age before checking Etag again
    post-check=0, pre-check=0
    (Discouraged) Old IE does not cache and always ask from server
  • Common usage
    Cache-Control: no-cache, max-age=315360000
    serve the cache for 1 year but if server version has updated, download from server
    public, max-age=2592000
    Tells the external caching system e.g. Varnish not to make a call to the web server for this request until the max-age header:cache-control:public max-age
    no-cache, must-revalidate, post-check=0, pre-check=0
    no cache and always ask from server
    max-age=0
    Chrome adds this in request when you enter the URL or hit refresh. This tells the cache server to serve the cache whenever possible. Whether or not the cache system serves the cache depends on the cache server. e.g. Varnish still serves the cache when it's possible

7.7.3. Expires header:expires

When the response is considered stale. Set to 0 means the response is expired. If Cache-Control has max-age or s-max-age, Expires is ignored.

7.7.4. Content-Security-Policy (CSP) - response

  • Content-Security-Policy is official name
    • X-Content-Security-Policy (Old Firefox and IE)
    • X-WebKit-CSP (Chrome and Safari)
  • https://developers.google.com/web/fundamentals/security/csp/
  • use Content-Security-Policy-Report-Only instead of Content-Security-Policy to test first!

  • A directive is to specify whitelist 

    Content-Security-Policy: script-src 'self' https://apis.google.com https://host1.com https://host2.com
  • is any other non-defined directives' default

  • Define multiple directives, 'none' means whitelist nothing (block anything) 

    Content-Security-Policy: default-src https://cdn.example.net; child-src 'none'; object-src 'none'
  • *://*.example.com:*

  • data: http: 

    script-src * data: https://ssl.gstatic.com 'unsafe-inline' 'unsafe-eval';
  • example.com means any scheme and any port
  • matches the current origin, but not its subdomains
  • allow inline JavaScript and CSS
  • allow text-to-JavaScript mechanisms like eval

By default, CSP bans inline script entirely if 'unsafe-inline', 'nonce-*' or 'sha256-' is not used

  • scripts inside <script> tags
  • inline event handlers and javascript: URLs

You will need to move content of script tags into an external file: 

<!--  change this -->
<script>
  function doAmazingThings() {
    alert('YOU AM AMAZING!');
  }
</script>
<button onclick='doAmazingThings();'>Am I amazing?</button>

<!-- To this -->
<!-- amazing.html -->
<script src='amazing.js'></script>
<button id='amazing'>Am I amazing?</button>

amazing.js 

function doAmazingThings() {
  alert('YOU AM AMAZING!');
}
document.addEventListener('DOMContentReady', function () {
  document.getElementById('amazing')
    .addEventListener('click', doAmazingThings);
});

report-uri directive tells the browser to POST JSON-formatted violation reports 

Content-Security-Policy: default-src 'self'; ...; report-uri /my_amazing_csp_report_parser;
// example
{
  "csp-report": {
    "document-uri": "http://example.org/page.html",
    "referrer": "http://evil.example.com/",
    "blocked-uri": "http://evil.example.com/evil.js",
    "violated-directive": "script-src 'self' https://apis.google.com",
    "original-policy": "script-src 'self' https://apis.google.com; report-uri http://example.org/my_amazing_csp_report_parser"
  }
}

Generate nonce value differently each time and insert it into any inline script, bash64 

<script nonce=EDNnf03nceIOfn39fn3e9h3sdfa>
  //Some inline code I cant remove yet, but need to asap.
</script>

Content-Security-Policy: script-src 'nonce-EDNnf03nceIOfn39fn3e9h3sdfa'

Or use sha 

<script>alert('Hello, world.');</script>
Content-Security-Policy: script-src 'sha256-qznLcsROx4GACP2dm0UCKCzCG-HiZ1guq6ZZDob_Tng='
// calculate sha hash for each inline script

To enable Google Analytics without 'unsafe-inline', move GA code in a js file and 

Content-Security-Policy: script-src www.google-analytics.com; img-src www.google-analytics.com

I've heard there's way other than 'unsafe-inline' for Google Tag Manager

referrer directive is like header:referrer-policy

7.7.5. X-Frame-Options

Whether the client can render the page in a <frame>, <iframe>, <object>.

DENY SAMEORIGIN :: the page can only be displayed in a frame on the same origin as the page itself. ALLOW-FROM uri :: Chrome doesn't support.

7.7.6. Content-Type

7.7.7. X-Content-Type-Options - response

Only one value
X-Content-Type-Options: nosniff
(no term)
It only applies to script and style
(no term)
Blocks a request if the requested type is

7.7.8. Content-Disposition

  • When used in email, it is more powerful. But in HTTP context, it has only 3 usage
    Content-Disposition: inline
    safe to be displayed in or as Web page
    (no term)
    Content-Disposition: attachment
    • or Content-Disposition: attachment; filename="filename.jpg"
    • Tells browser to download e.g. pop up `Save as` dialog
    (no term)
    Content-Disposition: form-data; name="fieldName"
  • php:rawurlencode

7.7.9. X-XSS-Protection - response

Not supported in FireFox. Use Content-Security-Policy for modern browsers. This is used for older browsers.

Disable XSS filtering
X-XSS-Protection: 0
Default, If XSS then the client will sanitize the page (remove the unsafe parts)
X-XSS-Protection: 1
Rather than sanitizing the page, the client will prevent rendering of the page if an attack is detected
X-XSS-Protection: 1; mode=block
Chrome only. If XSS is detected, the client will sanitize the page and report the violation. Same as report-uri in Content-Security-Policy
X-XSS-Protection: 1; report=<reporting-uri>

7.7.10. Referer - request header:referer

7.7.11. Referrer-Policy - response header:referrer-policy

Only works for Chrome, FireFox and Opera Controls how the client sends the Referer request header with requests that are made from your site. e.g. clicking on a link on your website and the link goes to either your website or other websites.

no-referrer-when-downgrade :: If no Referrer-Police is defined, clients should set to this. The url is sent as a referrer when the protocol security level stays the same HTTP->HTTP or HTTPS->HTTPS

no-referrer :: never set Referer header

same-origin :: only set Referer when the origin is the same.

origin :: always set the Referer header to the origin (domain) from which the request was made. Path information is stripped.

https://scotthelme.co.uk/blog1/ https://scotthelme.co.uk/blog2/ https://scotthelme.co.uk/
https://scotthelme.co.uk/blog1/ http://scotthelme.co.uk/blog2/ https://scotthelme.co.uk/
https://scotthelme.co.uk/blog1/ http://example.com/ https://scotthelme.co.uk/
     

strict-origin :: This value is similar to origin above but will not allow the secure origin to be sent on a HTTP request, only HTTPS.

origin-when-cross-origin :: The browser will send the full URL to requests to the same origin/domain but only send the origin when requests are cross-origin.

strict-origin-when-cross-origin :: Similar to origin-when-cross-origin above but will not allow any information to be sent when a scheme downgrade happens (the user is navigating from HTTPS to HTTP).

unsafe-url :: The browser will always send the full URL with any request to any origin.

7.7.12. Strict-Transport-Security - response

  • HSTS lets a website tell browsers that it should only be accessed using HTTPS. The first time your site is accessed using HTTPS and it returns the Strict-Transport-Security header, the browser records this information, so that future attempts to load the site using HTTP will automatically use HTTPS instead
  • Strict-Transport-Security: max-age=<expire-time>
    max-age
    The time, in seconds, that the browser should remember that a site is only to be accessed using HTTPS
  • Strict-Transport-Security: max-age=<expire-time>; includeSubDomains
    includeSubDomains Optional
    If this optional parameter is specified, this rule applies to all of the site's subdomains as well
  • Strict-Transport-Security: max-age=<expire-time>; preload
  • Applied in hosting:pantheon:hsts

7.7.13. Vary - response header:vary

  • Vary: <header-name>, <header-name>
  • Tell caching servers to deliver a different cache for different User-Agent
  • Tell caching servers to always deliver a refresh copy (all requests are uncacheable)
  • tell caching servers to deliver a different cache for different cookie header

7.7.14. Link header:link

  • Each link defined in HTML is separated by comma and values inside a link with attributes are delimited by ;
  • Refer to html:link
7.7.14.1. html:hreflang

7.7.15. Accept-CH - response header:accept-ch

Client Hints
Tell browsers to give the following headers for following requests:
  • DPR
  • Viewport-Width
  • actual width of an image in real physical pixels
  • boolean whether extra measures should be taken to reduce the payload

Initial page load has this in HTML 

<img src="flower.jpg" sizes="25vw">

sizes attribute has to be set in order for the Width header to be sent. In the near future, the width attribute on the image element is likely to be included in the algorithm too, but for now, we’ll have to stick with sizes. The sizes attribute describes the layout and display size of the image.

  • Header Width is then perfectly calculated.
  • Header DPR saves x in srcset and thus saves some markup.
  • deliver images which have approximate size
  • Use image server/proxy to deliver pin-point size picture
    • Base http://example.com/image.jpg
    • Actual url in HTML http://[key].lite.imgeng.in/http://example.com/image.jpg
    • Image server will deliver the images based on the CH request headers

7.8. MIME

  • Media type also known as Multipurpose Internet Mail Extensiions or MIME
  • https://www.iana.org/assignments/media-types/media-types.xhtml
  • rfc:mime:multipart/form-data
  • application/octet-stream
    • With header:content-disposition:attachment, browsers propose `Save as` dialog to download
    • Other application/*, browsers may propose to open with a matching software app/program
    • Default for binary files also means unknown binary file

    • PHP force a file to download 

      $fileNameAfterDownload = 'a.pdf';
$size = filesize($fn);
header('Content-Type: application/octet-stream');
header("Content-disposition: attachment; filename=\"{$fileNameAfterDownload}\"");

// Maybe unnecessary
header('Content-Transfer-Encoding:binary');

header("Content-Length:" . filesize($fn));

header("Expires:0");
header("Cache-Control:no-cache, must-revalidate");
header("Pragma:no-cache");

// This is to make sure we get a refresh copy of the source file
// e.g. if file_get_contents($fn) is run before this, and the file is modified after the 1st file_get_contents and before this, the 2nd file_get_contents in here gets the cached copy
clearstatcache();

$chunkSize = 25*(1024 * 1024); //25mb per chunky chunk

if ($size > $chunkSize) {
    $handle = fopen($fn, 'rb');
    while (!feof($handle)) {
        print(@fread($handle, $chunkSize));
        ob_flush();
        flush();
    }
    fclose($handle);
}
else {
    echo file_get_contents($fn);
}

die();

7.9. Cookie

HTTP response header 

Set-Cookie: <name>=<value>[; <Max-Age>=<age>]
[; expires=<date>][; domain=<domain_name>]
[; path=<some_path>][; secure][; HttpOnly]
httponly
client side script cannot access the cookie
secure
the client browser will prevent the transmission of a cookie over an unencrypted channel (non-https)
(no term)
SameSite
  • Same Site
    • www.a.ca is eq. to a.ca, and from a.ca to any subdomains is the same site
    • while from b.a.ca to c.a.ca is not same site
  • only action e.g. <a> links, cookies will be passed. Default since Feb 2020 for Chrome
  • requires secure
(no term)
Refer to setcookie, php:ini:session:cookie_secure, apache:mod_headers for adding flags to cookies
(no term)
A single cookie can hold less than 4KB. A lot of browsers restrict max 20 cookies for a domain
(no term)
Session relies on cookie usually sessionid (PHP default is to use PHPSESSID as cookie)
  • Session stores data on server side and hence can hold a lot of data
(no term)
Token vs session
  • token is stored on client side but generated by servers. Received by servers to validate permissions
  • session relies on both the client side's (JS, HTML, cookie) is served from the same server
  • while token can be used when front end and back end are served separately

7.10. Email

7.10.1. View email source on Outlook

Open the email, in Move tab under Message, Actions > Other Actions > View Source

7.10.2. Doctype

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html lang="en">
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <meta http-equiv="X-UA-Compatible" content="IE=edge" />
  <meta name="robots" content="noindex, nofollow">
  <link href='https://fonts.googleapis.com/css?family=Montserrat:400,700' rel='stylesheet' type='text/css'>

  <title>Product Update Hybrid</title>

  <style type="text/css">
    @import url(https://fonts.googleapis.com/css?family=Montserrat);
    body, table, td, a{-webkit-text-size-adjust: 100%; -ms-text-size-adjust: 100%;}
    table, td{mso-table-lspace: 0pt; mso-table-rspace: 0pt;}
    img{border: 0; height: auto; line-height: 100%; outline: none; text-decoration: none; -ms-interpolation-mode: bicubic;}
    table{border-collapse: collapse !important;}
    body{font-family: 'Montserrat', Arial, sans-serif; height: 100% !important; margin: 0 !important; padding: 0 !important; width: 100% !important;}

    /* https://github.com/seanpowell/Email-Boilerplate */

    div[style*="margin: 16px 0;"] { margin:0 !important; }

    a[x-apple-data-detectors] {
      color: inherit !important;
      text-decoration: none !important;
      font-size: inherit !important;
      font-family: inherit !important;
      font-weight: inherit !important;
      line-height: inherit !important;
    }
  </style>

  <!--[if mso]>
  <style type="text/css">
    .body-text {
      font-family: Arial, sans-serif !important;
    }
  </style>
  <![endif]-->

</head>

https://litmus.com/community/learning/24-how-to-code-a-responsive-email-from-scratch Use XHTML 1.0 Transitional 

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<meta name="format-detection" content="telephone=no"> 
<meta name="viewport" content="width=device-width; initial-scale=1.0; maximum-scale=1.0; user-scalable=no;">
<meta http-equiv="X-UA-Compatible" content="IE=9; IE=8; IE=7; IE=EDGE" />

    <title>Page title</title>

    <style type="text/css"> 
        @media screen and (max-width: 630px) {

        }
    </style>


</head>

<body style="padding:0; margin:0">

<table border="0" cellpadding="0" cellspacing="0" style="margin: 0; padding: 0" width="100%">
    <tr>
        <td align="center" valign="top">

        </td>
    </tr>
</table>

</body>
</html>

Style 

<style>
  @import url(http://fonts.googleapis.com/css?family=Roboto:300); /*Calling our web font*/

  /* Some resets and issue fixes */
body, table, td, a{-webkit-text-size-adjust: 100%; -ms-text-size-adjust: 100%;}
  table, td{mso-table-lspace: 0pt; mso-table-rspace: 0pt;}
  img{border: 0; height: auto; line-height: 100%; outline: none; text-decoration: none; -ms-interpolation-mode: bicubic;}
  table{border-collapse: collapse !important;}
  body{font-family: 'Montserrat', Arial, sans-serif; height: 100% !important; margin: 0 !important; padding: 0 !important; width: 100% !important;}

  #outlook a{ padding:0; }
  .ReadMsgBody{ width:100%; }
  .ExternalClass{ width:100%; }
  .backgroundTable{ margin:0 auto; padding:0; width:100%; !important; }
  .ExternalClass *{ line-height:115%; }
  /* End reset */

  /* These are our tablet/medium screen media queries */
  @media screen and (max-width:630px){

    /* Display block allows us to stack elements */
    *[class="mobile-column"]{ display:block; }
    /* Some more stacking elements */
    *[class="mob-column"]{ float:none !important; width:100% !important; }
    /* Hide stuff */
    *[class="hide"]{ display:none !important; }
    /* This sets elements to 100% width and fixes the height issues too, a god send */
    *[class="100p"]{ width:100% !important; height:auto !important; }
    /* For the 2x2 stack */
    *[class="condensed"]{ padding-bottom:40px !important; display:block; }
    /* Centers content on mobile */
    *[class="center"]{ text-align:center !important; width:100% !important; height:auto !important; }
    /* 100percent width section with 20px padding */
    *[class="100pad"]{ width:100% !important; padding:20px; }
    /* 100percent width section with 20px padding left & right */
    *[class="100padleftright"]{ width:100% !important; padding:0 20px 0 20px; }
    /* 100percent width section with 20px padding top & bottom */
    *[class="100padtopbottom"]{ width:100% !important; padding:20px 0px 20px 0px; }

  }
</style>

7.10.3. Guidelines

Campaign Monitor Mobile Design Guide

Element Note
table always assign align, width,border, cellpadding, cellspacing
tr <tr style="padding:0;margin:0;line-height:0;" height="15">
  <td>&nbsp;</td>
  </tr> // empty row with height
   
td Wrap text in td
   
Attribute  
style use display: none!important;
   

Inline text full width separator. Change attribute size and style height when needed. 

<hr noshade color="#FFFFFF" width="100%" size="1" 
    style="padding:0; margin:8px 0 8px 0; border:none; width:100%; height: 1px; color:#FFFFFF; background-color: #FFFFFF" />

<h2 style="font-family: 'Montserrat', Arial, sans-serif; font-size:20px; line-height:26px; color:#222222; font-weight:bold; text-transform:uppercase; padding:0 20px; margin:0;">You've been invited to this years event</h2>

7.10.4. Responsive

<style>
.sectionContainer {
  background-color: #ffffff;
  padding-right:10px !important;
  padding-left:10px !important;
}
.sectionTable {
  background-color: #ffffff;
}
@media only screen and (max-width:640px) {
.sectionTable {
  max-width:640px !important;
  width:100% !important;
}
}
@media only screen and (max-width:480px) {
.sectionTable {
  max-width:480px !important;
  width:100% !important;
}
.sectionContent .s480-w0 {
  display:none !important;
}
} 
</style>

<table>
  <tr>
    <td class="sectionContainer">
      <table border="0" cellpadding="0" cellspacing="0" width="640" class="sectionTable">
        <tr style="padding:0;margin:0;line-height:0;" height="15"><td>&nbsp;</td></tr>
        <tr>
          <td align="center" valign="middle" style="background-color:#004663;color:#ffffff;margin:0;font-size:24px;font-family:Georgia;font-weight:normal;">
            Section #1  
          </td>
        </tr>
        <tr style="padding:0;margin:0;line-height:0;" height="10"><td>&nbsp;</td></tr>
        <tr>
          <td align="center" valign="top">

          </td>
        </tr>
      </table>
    <td>
  </tr>
</table>

7.10.5. Even 3 columns

<td align="center" valign="top" style="font-size:0;">
  <!--[if (gte mso 9)|(IE)]>
  <table align="center" border="0" cellspacing="0" cellpadding="0" width="600">
    <tr>
      <td align="left" valign="top" width="190">
  <![endif]-->
  <div style="display:inline-block; max-width:33.3333%; min-width:190px; vertical-align:top; width:100%;" class="mobile-wrapper">

    <table align="left" border="0" cellpadding="0" cellspacing="0" width="100%" style="max-width:190px;" class="max-width">
      <tbody><tr>
        <td align="center" valign="top" style="font-family: Open Sans, Helvetica, Arial, sans-serif; padding-top: 25px;">

          <img src="icon-shield.png" width="50" height="50" border="0" style="display: block;">

          <h3 style="font-size: 18px; line-height: 24px;">Gmail</h3>
          <p style="color: #999999; font-size: 14px; line-height: 20px;">
            Mandeville carneiro robbins goas ross kelly ragan rodriguez stig jordan hodgekiss merlin yeaman
            <br><br>
            <a href="http://litmus.com" target="_blank" style="text-decoration: none; color: #75b6c9;">Read more →</a>
          </p>

        </td>
      </tr>
      </tbody></table>
  </div>
  <!--[if (gte mso 9)|(IE)]>
  </td>
  <td width="15" style="font-size: 1px;">&nbsp;</td>
  <td align="left" valign="top" width="190">
  <![endif]-->
  <div style="display:inline-block; max-width:33.3333%; min-width:190px; vertical-align:top; width:100%;" class="mobile-wrapper">
    <table align="center" border="0" cellpadding="0" cellspacing="0" width="100%" style="max-width:190px;" class="max-width">
      <tbody><tr>
        <td align="center" valign="top" style="font-family: Open Sans, Helvetica, Arial, sans-serif; padding-top: 25px;">

          <img src="icon-cloud-lock.png" width="50" height="50" border="0" style="display: block;">

          <h3 style="font-size: 18px; line-height: 24px;">Outlook</h3>
          <p style="color: #999999; font-size: 14px; line-height: 20px;">
            Mandeville carneiro robbins goas ross kelly ragan rodriguez stig jordan hodgekiss merlin yeaman
            <br><br>
            <a href="http://litmus.com" target="_blank" style="text-decoration: none; color: #75b6c9;">Read more →</a>
          </p>

        </td>
      </tr>
      </tbody></table>
  </div>
  <!--[if (gte mso 9)|(IE)]>
  </td>
  <td width="15" style="font-size: 1px;">&nbsp;</td>
  <td align="left" valign="top" width="190">
  <![endif]-->
  <div style="display:inline-block; max-width:33.3333%; min-width:190px; vertical-align:top; width:100%;" class="mobile-wrapper">
    <table align="right" border="0" cellpadding="0" cellspacing="0" width="100%" style="max-width:190px; float: right;" class="max-width">
      <tbody><tr>
        <td align="center" valign="top" style="font-family: Open Sans, Helvetica, Arial, sans-serif; padding-top: 25px;">

          <img src="icon-key.png" width="50" height="50" border="0" style="display: block;">

          <h3 style="font-size: 18px; line-height: 24px;">Lotus</h3>
          <p style="color: #999999; font-size: 14px; line-height: 20px;">
            Mandeville carneiro robbins goas ross kelly ragan rodriguez stig jordan hodgekiss merlin yeaman
            <br><br>
            <a href="http://litmus.com" target="_blank" style="text-decoration: none; color: #75b6c9;">Read more →</a>
          </p>

        </td>
      </tr>
      </tbody></table>
  </div>
  <!--[if (gte mso 9)|(IE)]>
  </td>
  </tr>
  </table>
  <![endif]-->
</td>

7.10.6. My even 2 columns

@media only screen and (max-width:480px){
      td[class="articleBlockRightColumn"]{
        text-align:center !important;
      }      
      td[class="articleBlockLeftColumn"],
      td[class="articleBlockRightColumn"]{
        display:block !important;
        width:100% !important;
        padding-bottom:10px;
      }
}

$html_default = [
  'left'  => 'articleBlockLeftColumn',
  'right' => 'articleBlockRightColumn',
  'ad'    => 'bigAds',
];
?>
<table border="0" cellpadding="0" cellspacing="0" width="100%">
  <tr>
    <td align="left" valign="top" width="50%" class="<?php echo $html['left']; ?>">
      <table border="0" cellpadding="0" cellspacing="0" width="100%">
        <tr>
          <td align="left" valign="top">
            <h3>
              <a href="<?php echo $item['url']; ?>" target="_blank" style="text-decoration: none;">
                <?php
                print theme('nc_custom_templates_inline_native_article',
                  ['article' => $item]
                );
                echo $item['title'];?></a>
            </h3>
            <p><?php echo $item['summary']; ?> <?php
              if ($readmore['inline']) {
                print theme(
                  'nc_custom_templates_inline_read_more',
                  ['item' => $item,
                   'options'=> $readmore
                  ]);
              }
               ?></p>
            <?php
              if (!$readmore['inline']) {
                print theme(
                  'nc_custom_templates_inline_read_more',
                  ['item' => $item,
                   'options'=> $readmore
                  ]);
              }
            ?>
          </td>
        </tr>
      </table>
    </td>

    <td align="right" valign="top" width="50%" class="<?php echo $html['right']; ?>">
      <table border="0" cellpadding="0" cellspacing="0" width="100%">
        <tr>
          <td align="right" valign="top" class="<?php echo $html['ad']; ?>">
            <a href="<?php echo $dfp['a']; ?>">
              <img src="<?php echo $dfp['img']; ?>" />
            </a>
          </td>
        </tr>
      </table>
    </td>
  </tr>
  <tr><td>&nbsp;</td></tr>
</table>

7.10.7. My template

Presets 

<?php
$_css_body = '-webkit-text-size-adjust:100%; -ms-text-size-adjust:100%;width:100% !important; margin:0;padding:0';
$_css_preheader = 'display: none; font-size: 1px; line-height: 1px; max-height: 0px; max-width: 0px; opacity: 0; overflow: hidden;';

$_css_section_column = 'background-color:#ffffff;padding-right:10px !important;padding-left:10px !important;';
// It's ok to set td with padding-right and left if td will not display:block when it's in mobile
// If td will display:block, then wrap content with <table><tr><td style="padding: 1em"></td></tr></table> to set padding.
$_css_section_table = 'max-width:640px;';
$_css_section_top = 'padding-top:15px;';

 $_css_text_small = 'font-size:10px;color:#555555;font-family:Arial,Helvetica,sans-serif !important;';
  $_css_p = 'font-size:16px;line-height: 20px !important;margin: 1em 0;font-family: Arial, Helvetica, sans-serif;';
  $_css_h1 = 'font-family:Arial,Helvetica,sans-serif !important; font-size:30px; line-height:36px; font-weight:bold; color:#ffffff;padding:0; margin:0;';
  $_css_h2 = 'font-family:Arial,Helvetica,sans-serif !important;font-size:20px;line-height:26px; font-weight:bold; color:#ffffff;padding:0; margin:0;';
  $_css_h3 = 'font-family:Arial,Helvetica,sans-serif !important;font-size:16px;line-height:22px;font-weight:bold;color:#333333;padding:0; margin:0;';
  $_css_h3_a = 'font-family:Arial,Helvetica,sans-serif !important;font-size:16px;line-height:22px;font-weight:bold;color:#333333;padding:0; margin:0;text-decoration: none;';
  $_css_h4 = 'font-family:Arial,Helvetica,sans-serif !important;font-size:14px;line-height:20px;font-weight:bold;color:#333333;padding:0; margin:0;';
  $_css_h4_a = 'font-family:Arial,Helvetica,sans-serif !important;font-size:14px;line-height:20px;font-weight:bold;color:#333333;padding:0; margin:0;text-decoration: none;';
?>
<style type="text/css">
td[class="articleBlockFullColumn"] {
    padding-bottom:10px !important;
}

@media screen and (max-width:640px) {

}

@media only screen and (max-width: 480px) {
p {
    padding-left: 0px !important;
    padding-right: 10px !important;
}
td[class="mobilecenter"] {
  text-align: center !important;
}
td[class="articleImg"],
td[class="articleDiv"] {
      display:none;
}

}
</style>

Global 

<body style="margin:0;padding:0;">
<table width="100%" height="100%" cellpadding="0" cellspacing="0" border="0" bgcolor="#F1F1F1">
  <tr>
    <td width="100%" valign="top" align="center">
      <div style="<?php echo $_css_preheader; ?>"><?php echo $_preview_text; ?></div>
      <center>
        <table width="100%" height="100%" cellpadding="0" cellspacing="0" border="0">
          <tr><td>Section 1</td></tr>
        </table>
      </center>
    </td>
  </tr>
</table>
</body>

Per section 

<tr>
  <td align="center" valign="top" style="<?php echo $_css_section_column; ?>">
    <table border="0" cellpadding="0" cellspacing="0" width="100%" style="<?php echo $_css_section_table; ?>">
      <tr>
        <td align="center" valign="top" style="<?php echo $_css_section_top; ?>">
          <table border="0" cellpadding="0" cellspacing="0" width="100%">
            <?php // section title ?>
            <tr>
              <td align="left" valign="top">
                <h2 style="<?php echo $_css_section_title; ?>">Press Releases</h2>
                <hr noshade color="#eb1f2a" width="100%" size="1"
                    style="padding:0; margin:8px 0 8px 0; border:none; width:100%; height: 1px; color:#eb1f2a; background-color: #eb1f2a" />
              </td>
            </tr>
            <?php // section content ?>
            <tr>
              <td align="left" valign="top">
                <?php for ($i = 0; $i < count($_item); $i++) {
                              $content_item = $_item[$i];
                              print theme(
                                'nc_custom_templates_component_full_width',
                                [
                                  'item'     => $content_item,
                'readmore' => $readmore,
                'html'     => $full_width_html,
                ]
                );
                }
                ?>
              </td>
            </tr>
          </table>
        </td>
      </tr>
    </table>
  </td>
</tr>

Section content 

<?php
/**
 * Created by PhpStorm.
 * User: Li Li
 * Date: 8/3/2017 2:28 PM
 *
 * $html => [
 *   'row' => 'articleBlockFullColumn',
 *   'img' => [
 * 'class' => 'articleImg'],
 *   'ad' => 'bigAds',
 * ]
 */

$html_default = [
  'row' => 'articleBlockFullColumn',
  'h3' => 'color:#333333;font-weight:bold;font-size:20px;font-family:Arial,Helvetica,sans-serif !important;',
  'h3 a' => 'text-decoration: none;color:#333333',
        'p' => 'line-height: 20px !important;margin: 1em 0;font-family: Arial, Helvetica, sans-serif;',
  'img' => [
    'enable'  => 1, // To show article thumbnail if it has one?
    'class'   => 'articleImg',
    'w'       => 140,
    'h'       => 105,
    'style'   => 'width:140px; height:105px; vertical-align:top;',
    ],
  'div' => [
    'class' => 'articleDiv',
    'w' => 15,
  ],
];
?>

<table border="0" cellpadding="0" cellspacing="0" width="100%">
  <tr>
    <td align="left" valign="top" width="100%" class="<?php echo $html['row']; ?>">
      <table border="0" cellpadding="0" cellspacing="0" width="100%">
        <tr>
          <?php if (!empty($item['image']) && $html['img']['enable']): ?>
            <td class="<?php echo $html['img']['class']; ?>" align="center" valign="top">
              <a
                href="<?php echo $item['url']; ?>"
                target="_blank"><img alt=""
                                     src="<?php echo $item['image']; ?>"
                                     width="<?php echo $html['img']['w']; ?>"
                                     height="<?php echo $html['img']['h']; ?>"
                                     style="<?php echo $html['img']['style']; ?>"/></a>
            </td>
            <td class="<?php echo $html['div']['class']; ?>"
                width="<?php echo $html['div']['w']; ?>">&nbsp;</td>
          <?php endif; ?>

          <td align="left" valign="top">
            <h3 style="<?php echo $html['h3']; ?>"><a
                href="<?php echo $item['url']; ?>"
                target="_blank"
                style="<?php echo $html['h3 a']; ?>"><?php
                print theme('nc_custom_templates_inline_native_article',
                  ['article' => $item]
                );
                echo $item['title'];
                ?></a></h3>
            <p style="<?php echo $html['p']; ?>"><?php echo $item['summary']; ?> <?php
              if ($readmore['inline']) {
                print theme(
                  'nc_custom_templates_inline_read_more',
                  ['item' => $item,
                   'options'=> $readmore
                  ]);
              }
              ?>
            </p>
            <?php
            if (!$readmore['inline']) {
              print theme(
                'nc_custom_templates_inline_read_more',
                ['item' => $item,
                 'options'=> $readmore
                ]);
            }
            ?>
          </td>
        </tr>
      </table>
    </td>
  </tr>
  <tr><td>&nbsp;</td></tr>
</table>

Table with 1 col wrapper 

$h_default = [
        'spacer' => '', // '' no spacer, top or bottom
        'content' => '',
        'align' => ['','right'], // first one is table then td
        'valign' => ['','top'],
        'class' => ['','mobilecenter'],
        'style' => ['',''],
];
?>
<table border="0" cellpadding="0" cellspacing="0" width="100%"
        <?php if ($h['align'][0]) {echo ' align="'.$h['align'][0].'"'; } ?>
        <?php if ($h['valign'][0]) {echo ' valign="'.$h['valign'][0].'"'; } ?>
        <?php if ($h['class'][0]) {echo ' class="'.$h['class'][0].'"'; } ?>
        <?php if ($h['style'][0]) {echo ' style="'.$h['style'][0].'"'; } ?>>
        <?php if (isset($h['spacer']) && $h['spacer'] == 'top'): ?>
                <tr><td>&nbsp;</td></tr>
        <?php endif; ?>
        <tr>
                <td <?php if ($h['align'][1]) {echo ' align="'.$h['align'][1].'"'; } ?>
                        <?php if ($h['valign'][1]) {echo ' valign="'.$h['valign'][1].'"'; } ?>
                        <?php if ($h['class'][1]) {echo ' class="'.$h['class'][1].'"'; } ?>
                        <?php if ($h['style'][1]) {echo ' style="'.$h['style'][1].'"'; } ?>><?php echo $h['content']; ?></td>
        </tr>
        <?php if (isset($h['spacer']) && $h['spacer'] == 'bottom'): ?>
                <tr><td>&nbsp;</td></tr>
        <?php endif; ?>
</table>

Table with 2 cols 

$_partnership_left  = theme(
  'nc_custom_templates_component_col_1',
  [
    'h' => [
      'spacer'  => '',
// '' no spacer, top or bottom
      'content' => "<a href='${_dfp_txt_1['href']}' target='_blank' style='ouline:none;display:inline-block;'><img src='${_dfp_txt_1['img']}' alt='' style='${_dfp_txt_1['img_style']}'></a>",
      'align'   => [ '', 'center' ],
// first one is table then td
      'valign'  => [ '', 'top' ],
      'class'   => [ '', 'mobilecenter' ],
      'style'   => [ '', 'padding: 0.4em' ],
    ],
  ]
);
$_partnership_right = theme(
  'nc_custom_templates_component_col_1',
  [
    'h' => [
      'spacer'  => '',
// '' no spacer, top or bottom
      'content' => "<h3 style='${_css_h3}'><a href='${_dfp_txt_1['href']}' target='_blank' style='${_css_h3_a}'>${_dfp_txt_1['title']}</a></h3>${_dfp_txt_1['content']}",
      'align'   => [ '', 'left' ],
// first one is table then td
      'valign'  => [ '', 'top' ],
      'class'   => [ '', '' ],
      'style'   => [ '', 'padding: 0.4em' ],
    ],
  ]
);
print theme(
  'nc_custom_templates_component_col_2',
  [
    'h' => [
      'table'   => [
        'align'  => '',
        'valign' => '',
        'class'  => '',
        'style'  => '',
      ],
      'spacer'  => '',
      'fixed'   => FALSE,
      'content' => [
        $_partnership_left,
        $_partnership_right
      ],
      'align'   => [ 'center', 'left' ],
      'valign'  => [ 'top', 'top' ],
      'class'   => [
        'partnershipBlockContentLeft',
        'partnershipBlockContentRight'
      ],
      'style'   => [ '', '' ], // padding: 0.4em
    ],
  ]
);

7.10.8. Hybrid Responsive Design

Orignal responsive design: parent table width:100%, child table.responsive-table width:600 and use media queries 

<table border="0" cellpadding="0" cellspacing="0" width="100%"> 
    <tr>
        <td bgcolor="#00a9f7" align="center">
            <table border="0" cellpadding="0" cellspacing="0" width="600" class="responsive-table">                
                <tr>
                    <td align="center" valign="top" style="padding: 40px 0px 40px 0px;">
                        <!-- IMAGE -->
                        <img alt="Example" src="http://placehold.it/600x300" width="600" style="display: block;" border="0" class="responsive-image">
                    </td>
                </tr>
                <tr>
                    <td align="center" valign="top" style="padding: 0px 10px 20px 10px;">
                        <!-- HEADLINE -->
                        <p style="color: #ffffff; font-family: sans-serif; font-size: 24px; font-weight: bold; line-height: 28px; margin: 0;">Announcing Some News</p>
                    </td>
                </tr>
                <tr>
                    <td align="center" valign="top" style="padding: 0px 10px 60px 10px;">
                        <!-- COPY -->
                        <p style="color: #b5e2f7; font-family: sans-serif; font-size: 16px; font-weight: normal; line-height: 24px; margin: 0;">Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim.</p>
                    </td>
                </tr> 
            </table>
        </td>
    </tr>
</table>

@media screen and (max-width: 600px) {
  .responsive-table {
    display: block;
    width: 100% !important;
  }

  .responsive-image {
    height: auto;
    max-width: 100% !important;
  }
}

Hybrid or spongy design: parent table width:100%, child table width:100% with max-width:600px, for IE, wrap another table width:600 around the child table. 

<table border="0" cellpadding="0" cellspacing="0" width="100%"> 
    <tr>
        <td bgcolor="#00a9f7" align="center">
            <!--[if (gte mso 9)|(IE)]>
            <table align="center" border="0" cellspacing="0" cellpadding="0" width="600">
            <tr>
            <td align="center" valign="top" width="600">
            <![endif]-->
            <table border="0" cellpadding="0" cellspacing="0" width="100%" style="max-width: 600px;" >                
                <tr>
                    <td align="center" valign="top" style="padding: 40px 0px 40px 0px;">
                        <!-- IMAGE -->
                        <img alt="Example" src="http://placehold.it/600x300" width="600" style="display: block; width: 100%; max-width: 100%;" border="0">
                    </td>
                </tr>
                <tr>
                    <td align="center" valign="top" style="padding: 0px 10px 20px 10px;">
                        <!-- HEADLINE -->
                        <p style="color: #ffffff; font-family: sans-serif; font-size: 24px; font-weight: bold; line-height: 28px; margin: 0;">Announcing Some News</p>
                    </td>
                </tr>
                <tr>
                    <td align="center" valign="top" style="padding: 0px 10px 60px 10px;">
                        <!-- COPY -->
                        <p style="color: #b5e2f7; font-family: sans-serif; font-size: 16px; font-weight: normal; line-height: 24px; margin: 0;">Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim.</p>
                    </td>
                </tr> 
            </table>
            <!--[if (gte mso 9)|(IE)]>
            </td>
            </tr>
            </table>
            <![endif]-->
        </td>
    </tr>
</table>

Fab Four Technique to create responsive emails without media queries

  • If width is greater than max-width, max-width wins
  • If min-width is greater than width or max-width, min-width wins.
  • Demo

So the following sets the box to 480px 

.box {
    width:320px;
    min-width:480px;
    max-width:160px;
}

Use calc() to set width. The following creates 2 columns that will stack and grow below 480px 

.block {
    display:inline-block;
    min-width:50%;
    max-width:100%;
    width:calc((480px — 100%) * 480);
}

calc() is only not supported in latest Outlook for Windows, OWA both Office 365 and Outlook.com, and Yahoo webmail and iOS/Android apps. In these cases 

.block {
    display:inline-block;
    min-width:240px;
    width:50%;
    max-width:100%;
    min-width:calc(50%);
    width:calc(480px * 480 — 100% * 480);
}

Final version with fixes for OWA and WebKit Prefixes 

.block {
    display:inline-block;
    min-width:240px;
    width:50%;
    max-width:100%;
    min-width:-webkit-calc(50%);
    min-width:calc(50%);
    width:-webkit-calc(230400px — 48000%);
    width:calc(230400px — 48000%);
}

7.10.9. Inbox Preview

Litmus
covers the most email clients and used by MailChimp and Campaign Monitor. Plus plan has Live Edit and Live Preivew
Email on Acid
Basic plan covers unlimited tests. Fewer Android devices are supported compared with Litmus
Testi@
less email clients but much cheaper. Has Chrome Extension

7.10.10. Gmail Image Caching

Images are downloaded at proxy server and serve those images from that proxy instead. One download only. Affects stats of opens and geolocation. Gmail Web and Gmail iOS and Android apps.

7.10.11. CSS and HTML Support across Devices email:css:support

  • https://www.campaignmonitor.com/css/
  • https://templates.mailchimp.com/resources/email-client-css-support/
  • https://www.caniemail.com
  • Gmail Web doesn't support <style> in <body> but supports <style> in <head>
  • Gmail removes support for CSS attribute selectors
  • not supported in
    Browsers
    Outlook.com, Yahoo!, Gmail, AOL
  • <style> in <head> is not supported in:
    • Gmail Android app IMAP, Gmail mobile webmail, Yahoo! Mail Android app
    • AOL Mail
  • not supported in
    Desktop
    Outlook 2007-16, Windows 10 Mail
    Mobile
    Gmail Android app IMAP, Gmail mobile wemail but supports in Gmail Android app, Gmail iOS app
    Webmail
    AOL Mail, Outlook.com
    Don't use and or!
    some email clients don't support them e.g. Yahoo! Mail
  • not supported in
    Mobile
    Gmail Android app IMAP, Gmail mobile webmail
  • only supported in
    Desktop
    Outlook 2000-03, Apply Mail 10, Outlook for Mac
    Mobile
    Android Mail 4.4.4, iOS 10 Mail, iOS 11 Mail, Outlook Android app, Sparrow
    Webmail
    AOL Mail
  • not supported in
    • Outlook 2007-16
    • Outlook Express
    • Windows 10 Mail
    • Windows Live Mail
    • Android 4.2.2 Mail
    • Gmail Android app IMAP
    • Windows Phone 8 Mail
    • Yahoo! Mail Android app
  • Email client safe fonts
    • Courier
    • Courier New
    • Georgia
    • Times
    • Times New Roman
  • Refer to css:font-face:web safe fonts for checking font coverage and font stack

Target iPhone 5 and above mjml:tag:mj-hero:sample1 

@media all and (max-width: 414px) {
  table[style*="color:#123456;"] { width:100%;}
  td[style*="font-family:Khand, Helvetica, sans-serif, Arial;font-size:18px;line-height:28px;color:#666465;"] {
    color: #000000 !important;
    background-color: rgba(255,255,255,0.5) !important;
  }
  tr.spacer-above-button {
     height:50px;
  }
}

@media screen and (max-device-width: 320px) and (max-device-height:
568px) { /* iPhone 8 (Zoom View) */ }

@media screen and (max-device-width: 375px) and (max-device-height:
667px) { /* iPhone 8 (Standard View) and iPhone 8 Plus (Zoom View) */ }

@media screen and (max-device-width: 414px) and (max-device-height:
776px) { /* iPhone 8 Plus (Standard View) */ }

Prevent auto-scaling on iOS 11 

<meta name="x-apple-disable-message-reformatting">

Target Samsung Mail 

#MessageViewBody .yourclassnamehere { }

7.10.12. MJML

7.10.12.1. Install & Basics
npm install mjml

# you can delete node_modules folder and reinstall to get the latest version

./node_modules/.bin/mjml -V

# To avoid typing ./node_modules/bin/
export PATH="$PATH:./node_modules/.bin"

# mjml version (e.g. 3.3.5)
mjml -V

# one time output html
mjml input.mjml

# one time output to a specific file
mjml input.mjml -o my-email.html

# watch a file
mjml -w input.mjml

# convert v3 .mjml to v4 .mjml
mjml -m index.mjml indexv4.mjml

# watch multiple
# html files will be in views/**/*.html
mjml -w source/**/*.mjml -o views

# minify html output
# Don't use as it will break Outlook previews
node_modules/.bin/mjml path/to/index.mjml -o path/to/index.html --config.minify
  • Since v4, install Windows App https://mjmlio.github.io/mjml-app/ to get WYSIWYG experience
  • Install Litmus Chrome Extension and follow the instructions to tweak the extension setting. Login to Litmus.com and open a local .html file
  • Install Atom packages to edit .mjml in Atom: linter-mjml and mjml-preview
  • Minify html as some email clients e.g. Gmail App on iOS and Android clips the first 102kb HTML to convert and display
  • Tags or mj-elements styling
    • a limited set of attributes can be used
    • For now, must repeat for each tag the style you want to apply on it
    • Another way is to create custom component, sub-classing a standard one and have your styles defined by default

Sample 1

7.10.12.2. mjml mjml:tag:mjml
  • Contains mjml:tag:mj-head and mjml:tag:mj-body
  • Attributes
    owa
    Default none, display the mobile version for Outlook.com. To force the desktop layout on Outlook.com <mjml owa="desktop">
    lang
    Default none. Set <html lang="">
7.10.12.3. mj-head mjml:tag:mj-head

Contains styles and meta elements. To insert custom head elements, registerMJHeadElement(<string> name, <funciton> handler) api from mjml-core

7.10.12.4. mj-head > mj-preview mjml:tag:mj-preview

no other attributes 

<mj-head>
    <mj-preview>Hello MJML</mj-preview>
</mj-head>
7.10.12.5. mj-head > mj-attributes mjml:tag:mj-attributes
  • mj-all sets default attributes for every component inside your MJML document
  • inline attributes > classes > default mj-attributes > defaultMJMLDefinition
<mjml>
  <mj-head>
    <mj-attributes>
      <mj-text padding="0" />
      <mj-class name="blue" color="blue" />
      <mj-class name="big" font-size="20px" />
      <mj-all font-family="Arial" />
      <mj-all padding="0px"></mj-all>
    </mj-attributes>
  </mj-head>
  <mj-body>
    <mj-container>
      <mj-section>
        <mj-column>
          <mj-text mj-class="blue big">
            Hello World!
          </mj-text>
        </mj-column>
      </mj-section>
    </mj-container>
  </mj-body>
</mjml>
7.10.12.6. mj-head > mj-breakpoint

On which breakpoint the layout should go desktop/mobile. Default is 480px 

<mjml>
  <mj-head>
    <mj-breakpoint width="320px" />
  </mj-head>
  <mj-body>
    <mj-section>
      <mj-column>
        <mj-text>
          Hello World!
        </mj-text>
      </mj-column>
    </mj-section>
  </mj-body>
</mjml>
7.10.12.7. mj-head > mj-style mjml:tag:mj-style
  • Refer to email:css:support
  • mj-style
  • inline="inline"

    • When inline="inline" used, don't use double quotes inside mj-style 

        <mjml>
<mj-head>
  <mj-style>
    @media (max-width: 480px) { div[style*="color:#F45e46;"] { text-align: center !important } }
  </mj-style>
  <mj-style inline="inline">
    .link-nostyle { color: inherit; text-decoration: none }
    <!-- for a tag, color: inherit is not working, either specify color in .link-nostyle or style="color:#fff" in a tag. -->
  </mj-style>
</mj-head>
<mj-body>
  <mj-container>
    <mj-section>
      <mj-column>

    <mj-image width="100" src="/assets/img/logo-small.png"></mj-image>

    <mj-divider border-color="#F45E43"></mj-divider>

    <mj-text font-size="20px" color="#F45e46" font-family="helvetica">
      Hello <a href="https://mjml.io" class="link-nostyle">World</a>
    </mj-text>

      </mj-column>
    </mj-section>
  </mj-container>
</mj-body>
  </mjml>

  • mjml:tag:mj-style:responsive font size 

    <mjml>
  <mj-head>
 <mj-style>
   .body-fix div {
      font-size: 22px !important; /* This will override font-size attribute if client supports style in head.*/
      /* Define styles use mjml elements' attribute for devices don't support <style> in <head>:
    Gmail Android app IMAP, Gmail mobile webmail, Yahoo! Mail Android app, Google Inbox, Android 5.1+ Native Mail App
      */
      line-height: 28px !important;
    }
    @media only screen and (max-width: 480px){
      .body-fix div {
    font-size: 28px !important;
    line-height: 44px !important;
      }
    }
</mj-style>
  </mj-head>
  <mj-body>
<mj-container>
  <mj-section>
    <mj-column>
      <mj-text css-class="body-fix" font-size="20px" color="#F45E43" font-family="helvetica">Will work :)</mj-text>
    </mj-column>
  </mj-section>
</mj-container>
  </mj-body>
</mjml>

  • Use css-class in any mj-element and change styles for mobile e.g. hide on mobile 

    <mj-style>
  @media all and (max-width:480px) {
tr[class*=hidden] { display:none; }
<!-- Better -->
*[class*=hidden] { display:none !important;}
  }

  <!-- Hide column on mobile:
   - Desktop clients: all ok
   - Mobile: only Gmail IMAP app displays the column while it should not show
   - Web browser: all ok
   -->
  .hide-on-mobile {display:none !important;}
  @media (min-width:480px) {
.hide-on-mobile { display: inline-block !important;}
  }

  <!-- Hide column on mobile: My Version as the above solution is hard to comprehend
   - Desktop clients: all ok
   - Mobile: only Gmail IMAP app displays the column while it should not show
   - Web browser: all ok
-->
  .hide-on-mobile {display:none !important;}
  .hide-on-mobile-outlook .hide-on-mobile {display: inline-block !important;}
  @media (min-width:480px) {
.hide-on-mobile {display:inline-block !important;}
.hide-on-mobile-outlook .hide-on-mobile {display: inline-block !important;}
  }
</mj-style>

<!-- Show on mobile only, hide on desktop
  - Desktop clients: all hidden (good!)
  - Mobile clients: only Gmail App IMAP does not show, all mobile clients show (good!)
  - Web browser: all hidden (Good!)
-->
<mj-style inline="inline">
  .show-on-mobile-only, .show-on-mobile-only-outlook {display:none;}
  .show-on-mobile-only table, .show-on-mobile-only-outlook table {mso-hide:all;}
</mj-style>
<mj-style>
  @media (max-width:480px) {
.show-on-mobile-only, .show-on-mobile-only-outlook { display: inline-block !important;}
  }
</mj-style>
<!-- Show on mobile only, hide on desktop. -->

<mj-text css-class="hidden"></mj-text>
<mj-column css-class="hide-on-mobile"></mj-column>

<mj-section css-class="show-on-mobile-only"></mj-section>

    • css-class only works for mj-* elements, use this to target elements inside mj-text 

        <mj-style>
    @media all and (max-width: 361px) { 
      h2[class*="header"] { font-size:50px!important;}
    }
    @media all and (max-width: 321px) {
      h2[class*="header"] { font-size:49px!important;}
    }
  </mj-style>
  <!-- 
   h2[class*="header"] { font-size:55px!important;}
  -->

  <mj-text css-class="main-content" color="#001f58" padding="0px 0px 0px 0px">
<h2 css-class="main-content" class="header" style="font-size:55px;line-height:50px;font-family:'Trebuchet MS',sans-serif;padding:0;margin:0;font-weight:100;padding:0;margin:0;">...</h2>
  </mj-text>
7.10.12.8. mj-head > mj-font mjml:tag:mj-font
<mjml>
  <mj-head>
    <mj-font name="Raleway" href="https://fonts.googleapis.com/css?family=Raleway" />
  </mj-head>
  <mj-body>
    <mj-container>
  <mj-section>
    <mj-column>
      <mj-text font-family="Raleway, Arial">
        Hello World!
      </mj-text>
    </mj-column>
  </mj-section>
    </mj-container>
  </mj-body>
</mjml>
7.10.12.9. mj-head > mj-title mjml:tag:mj-title
<mj-head>
  <mj-title>Hello MJML</mj-title>
</mj-head>
7.10.12.10. mj-head > mj-body > mj-section (mj-container)

mjml:tag:mj-body mjml:tag:mj-section mjml:tag:mj-container

  • To insert custom elements either registerMJElement(<MJMLElement> class) api from mjml-core or via .mjmlconfig file
  • Non-known element from mjml-core are ignored
  • can have multiple mj-section. For v3, mj-body should have only one root element which usually is mj-container, which contains the entire content
    • Attributes
      width
      default 600px
      background-color
      default n/a. Don't set to white as email clients may use Dark Mode
      (no term)
      css-class
  • There's no more mj-container in version 4. mj-section was mj-container in version 3 and mj-container was mj-body in version 1.x
    • Attributes
      full-width
      default n/a. If set to full-width then section is extended to the full width beyond 600px
      (no term)
      background-color
      (no term)
      background-url
      background-size
      auto
      background-repeat
      repeat
      vertical-align
      top
      text-align
      center
      padding
      20px 0
      direction
      ltr (or rtl)
    • rendered HTML is class="xx"
7.10.12.11. mj-body > mj-include mjml:tag:mj-include

Inside mjml > mj-body > mj-container <mj-include path="./header" /> <!– or 'header.mjml' –>

7.10.12.12. mj-body > mj-section mjml:tag:mj-section
  • Should contain mj-column. Up to 4 columns
  • Section has up to 600px width
  • May use % for left and right padding
  • Structure the columns in the order you want them to stack on mobile, then use direction attribute to change the order on desktop
  • Refer to https://mjml.io/documentation/#mjml-section
    full-width
    n/a (full-width)
    direction
    ltr or rtl, default ltr
    padding
    default 20px 0
    (no term)
    padding-top, padding-x
    background-url
    default n/a
    background-repeat
    default repeat, no-repeat
    background-color
    default n/a,
7.10.12.13. mj-wrapper > mj-section mjml:tag:mj-wrapper
  • Wrap multiple sections. Perfect place to set borders
  • Refer to https://mjml.io/documentation/#mjml-wrapper
    padding
    20px 0
    (no term)
    padding-*
    background-url
    If mj-wrapper has this defined, don't define in its mj-section
    background-repeat
    repeat
    background-size
    auto
    border
    none
    direction
    ltr (rtl)
    vertical-align
    top
    text-align
    center
    full-width
    Don't add full-width="full-width" inside mj-section as it will make mj-section non-full-width
    • I also find out if there are other sections that are also inside mj-container, define full-width only for that wrapper and don't define it for other sections
    • Sometimes mj-section does not have to be full-width, you can set left/right padding to 0 for mj-section
    (no term)
    css-class
<mjml>
  <mj-body>
    <mj-section></mj-section>
    <mj-wrapper border="1px solid #000000" padding="50px 30px">

  <mj-section border-top="1px solid #aaaaaa" border-left="1px solid #aaaaaa" border-right="1px solid #aaaaaa" padding="20px">
    <mj-column>
      <mj-image padding="0" src="https://placeholdit.imgix.net/~text?&w=350&h=150" />
    </mj-column>
  </mj-section>

  <mj-section border-left="1px solid #aaaaaa" border-right="1px solid #aaaaaa" padding="20px" border-bottom="1px solid #aaaaaa">
    <mj-column border="1px solid #dddddd">
      <mj-text padding="20px"> First line of text </mj-text>
      <mj-divider border-width="1px" border-style="dashed" border-color="lightgrey" padding="0 20px" />
      <mj-text padding="20px"> Second line of text </mj-text>
    </mj-column>
  </mj-section>

    </mj-wrapper>

  </mj-body>
</mjml>
  1. mj-container > mjml:tag:mj-column

    Any mj-element included in a column will have a width equals to 100% of this column's width. e.g. mj-text

    Inside a column, any standard element, or the ones you've defined and registered can be included.

    But don't include another column or section.

    background-color
    default n/a
    1. Auto sizing

      2 columns, each column 50%, 3 columns each is 33%, etc.

    2. Manual sizing
      <mj-column width="300px">
 <!-- First column content -->
</mj-column>
<mj-column> 
 <!-- Second column content -->
</mj-column>

      Once one column's width is set, you have to set manually each column size.

      Always keep in mind that the maximum space to end up with a responsive layout is 600px.

      vertical-align
      default top, can be middle/bottom
      width
      default (100/number of columns %)

      No padding is set by default

7.10.12.14. mj-container > mj-hero mjml:tag:mj-hero

https://mjml.io/documentation/#mjml-hero 

<mj-container>
  <mj-hero mode="fluid-height" background-width="600px" background-height="469px" background-url="https://via.placeholder.com/600x469/?text=600x469" background-color="#2a3448" padding="100px 0px">
    <!-- To add content like mj-image, mj-text, mj-button ... use the mj-hero-content component -->
    <mj-hero-content width="100%">
      <mj-text padding="20px" color="#ffffff" font-family="Helvetica" align="left" font-size="45" line-height="45px" font-weight="900">
        GO TO SPACE
      </mj-text>
      <mj-button href="https://mjml.io/" align="left">
        ORDER YOUR TICKET NOW
      </mj-button>
    </mj-hero-content>
  </mj-hero>
</mj-container>

Always specify background-color mode :: fluid-height or fixed-height 

<mj-container>
  <mj-hero mode="fixed-height" height="469px" background-width="600px" background-height="469px" background-url="https://via.placeholder.com/600x469/?text/600x469" background-color="#2a3448" padding="100px 0px">
    <!-- To add content like mj-image, mj-text, mj-button ... use the mj-hero-content component -->
    <mj-hero-content width="100%">
      <mj-text padding="20px" color="#ffffff" font-family="Helvetica" align="center" font-size="45" line-height="45px" font-weight="900">
        GO TO SPACE
      </mj-text>
      <mj-button href="https://mjml.io/" align="center">
        ORDER YOUR TICKET NOW
      </mj-button>
    </mj-hero-content>
  </mj-hero>
</mj-container>

mj-hero can be a mj-section mjml:tag:mj-hero:sample1 

<mj-hero
  mode="fluid-height"
  height="420px"
  background-width="600px"
  background-height="420px"
  background-url="http://600x420.jpg"
  background-color="#FFFFFF"
  padding="0px">
  <mj-hero-content width="100%">
    <mj-text font-size="18px" font-family="Khand, Helvetica, sans-serif, Arial" font-weight="900">
  Title aligned to left
    </mj-text>
    <mj-table width="55%" color="#123456">
  <tr style="list-style: none;line-height:1">
    <td style="font-family:Khand, Helvetica, sans-serif, Arial;font-size:18px;line-height:28px;color:#666465;">
      This text aligned to the left and takes 55% width
    </td>
  </tr>
    </mj-table>
    <mj-spacer height="20px" css-class="spacer-above-button" />
    <mj-button background-color="#FF0025" font-size="16px" align="left" font-family="'Merriweather Sans', Helvetica, sans-serif, Arial" padding-left="25px">
  Learn more
    </mj-button>
    <mj-spacer height="20px" />
    <mj-text font-size="14px" font-weight="900" font-family="'Merriweather Sans', sans-serif;">
  Phone number&nbsp;&nbsp;&nbsp;<a href="http://a.com" style="color:#000000;text-decoration:none;">a.com</a>
    </mj-text>
    <mj-social color="#333333" align="left" text-mode="false" icon-size="16px" padding-left="25" padding-bottom="10" padding-top="10" mode="horizontal" display="linkedin:url facebook:url twitter:url youtubec:url" linkedin-href="https://www.linkedin.com/"
       linkedin-content=" "
       linkedin-icon-color="#A7A6A4" facebook-href="https://www.facebook.com/" facebook-content=" "
       facebook-icon-color="#A7A6A4"
       twitter-href="https://twitter.com/"
       twitter-content=" "
       twitter-icon-color="#A7A6A4" youtubec-href="https://www.youtube.com/"
       youtubec-content=" "
       youtubec-icon-color="#A7A6A4" youtubec-icon="http://16x16.jpg" />
    <mj-spacer height="30px" />
  </mj-hero-content>
</mj-hero>
<mj-section>
...
</mj-section>
7.10.12.15. mj-section > mj-column mjml:tag:mj-column
  • Attributes
    width
    without it, mj-column is evenly distributed. Can be pixels or percentage
    vertical-align
    top
    padding, padding-*
    20px 0
    css-class
    border, border-*
    backgrond-color
7.10.12.16. mj-elements inside mj-column
  • Any mj-element inside mj-column will have width equivalent to 100% of this column's width
  • Any mj-element or elements inside mj-raw including mj-image should not have a higher width than the current column, otherwise Outlook will have troubhle in laying out mj-column
  1. mj-text mjml:tag:mj-text
    • It can contain any HTML tag with any attributes
    • 10px 25px
    • #000000
    • n/a
    • Ubuntu, Helvetica, Arial, sans-serif
    • 13px
    • font-style
    • 22px. Use %.
    • none
    • left

    • Basic 

      <mj-section background-color="#f0f0f0">
  <mj-column>
<mj-text font-style="italic" font-size="20" color="#626262">
   My Company <a href="#" style="text-decoration:none!important;text-decoration:none;color:#2DDCB4">... read more</a>
</mj-text>
<mj-text>
  <h1>Hello Title</h1>
</mj-text>
 </mj-column>
</mj-section>
    • mjml:tag:mj-style:responsive font size
    • https://litmus.com/blog/update-banning-blue-links-on-ios-devices
      • <meta content="telephone=no" name="format-detection">
      • 6&zwnj;75 Massachusetts Ave, Cambridge, MA 02139 is 675 Massachusetts Ave, Cambridge, MA 02139
      • a[href^=tel]{ color:#F00; text-decoration:none;}
    • Change styles for all iOS data detection!
      iOS add an attribute to all links
      <a href="#" x-apple-data-detectors="true">
      (no term)

      So add style 

         a[x-apple-data-detectors] {
   color: inherit !important;
   text-decoration: none !important;
   /* below is optional */
   font-size: inherit !important; 
   font-family: inherit !important;
   font-weight: inherit !important;
   line-height: inherit !important;
   }

   // Samsung Mail automatically adds id MessageViewBody to <body> and a <div id="MessageWebViewDiv">
   #MessageViewBody a {
color: inherit;
text-decoration: none;
font-size: inherit;
font-family: inherit;
font-weight: inherit;
line-height: inherit;
   }
    • Customize line-height for Outlook desktop clients only
      • Define ling-height in mj-text attribute and then set @media
      • line-height for Outlook desktop clients is 95% while others support @media use 83%

      • This can be used to target other CSS property for Outlook desktop clients only 

          <mj-style>
  @media all and (min-width: 500px) {
.main-content-h1-lh div {
  line-height:83%!important;
}
  }
  </mj-style>
  <mj-text css-class="main-content-h1-lh" line-height="95%">...</mj-text>
  2. mj-button, Image header with text and button
    href
    link has to be valid!
    target
    _blank
    padding
    10px 25px
    inner-padding
    10px 25px
    background-color
    #414141
    container-background-color
    n/a
    color
    #ffffff
    border
    none
    border-radius
    3px
    font-size
    13px
    font-weight
    normal
    font-family
    Ubuntu, Helvetic, Arial, sans-serif
    line-height
    120%, px/%/none
    align
    center
    vertical-align
    middle
    text-align
    none
    text-decoration
    none, underline/overline/none
    text-transform
    none, capitalize/uppercase/lowercase
    rel
    link rel attribute
    (no term)
    css-class
    <mj-section background-url="http://1.bp.blogspot.com/-TPrfhxbYpDY/Uh3Refzk02I/AAAAAAAALw8/5sUJ0UUGYuw/s1600/New+York+in+The+1960's+-+70's+(2).jpg" background-size="cover" background-repeat="no-repeat">
  <!-- -In order to have the background rendered full-width in the column, 
  set the column width to 600px-->
  <mj-column width="600">

    <mj-text align="center" color="#fff" font-size="40" font-family="Helvetica Neue">Slogan here</mj-text>

    <mj-button background-color="#F63A4D" href="#">
  Promotion
    </mj-button>

  </mj-column>

</mj-section>
  3. mj-image mjml:tag:mj-image
    • Attributes
      width

      px, default 100% (100% of the column width)

      • mj-image width is is 300px, mj-image left/right total padding is 0px, column width of mj-image is 690*0.5=345px
      • On desktop, image display width is min[ Column_Width - MJ_Image_Paddings, MJ_Image_Width ] = min[ 345-0=345, 300 ] = 300px
      • On mobile, image display width is min[ 100% vw - 50, Column_Width - MJ_Image_Paddings, MJ_Image_Width ] = 300x

      • works on all clients except Yahoo! Mail on web 

        <mj-style inline="inline">
  .img-300x169 {width:300px;height:169px;}
</mj-style>

<mj-image css-class="img-300x169" width="300px" height="169px" src="..." alt="" />
      height
      auto
      align
      center
      padding
      10px 25px
      (no term)
      padding-*
      border
      none
      border-width
      default 4px
      border-style
      default solid, dashed/dotted
      border-color
      default #000000
      alt
      string
      (no term)
      href
      target
      _blank
      srcset
      only supported in iOS. Determine display size and multiply the dimension by 2

    When mj-image is inside a mj-column that is the only column, then size doesn't matter. If mj-image is inside a column that has other sibling columns, then limit width so that columns are layed out correctly in Outlook. 

    <mj-section background-color="white">

  <!-- Left image -->
  <!-- column is 50% of total width,  mj-image is vertically and horizontally centered and image width is always 200 -->
  <mj-column>
    <mj-image width="200px" src="http://via.placeholder.com/300x400/?text=300x400" />
  </mj-column>

  <!-- right paragraph -->
  <mj-column>
    <mj-text font-style="italic" font-size="20" font-family="Helvetica Neue" color="#626262">
  Find amazing places
    </mj-text>

    <mj-text color="#525252">
  Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin rutrum enim eget magna efficitur, eu semper augue semper. Aliquam erat volutpat. Cras id dui lectus. Vestibulum sed finibus lectus.</mj-text>

  </mj-column>
</mj-section>
  4. mj-social mjml:tag:mj-social
    • Refer to mjml:tag:mj-table:social

    • Version 4 

      <mj-section background-color="#e7e7e7">
  <mj-column>
<mj-social mode="horizontal">
  <mj-social-element name="facebook" />
  <mj-social-element name="twitter-noshare" href="..." src="..." alt="" background-color="#3d3d3d" />
</mj-social>
  </mj-column>
</mj-section>
      mj-social
      attributes
      align
      center
      border-radius
      3px
      font-family
      Ubuntu, Helvetica, Arial, sans-serif
      font-size
      13px
      icon-size
      20px. percent/px.
      icon-height
      20px. percent/px. Overrides icon-size
      icon-padding
      0px
      line-height
      22px
      inner-padding
      4px
      padding
      10px 25px
      container-background-color
      n/a
      mode
      horizontal (vertical)
      text-decoration
      none
      text-padding
      4px 4px 4px 0
      color
      #333333
    • attributes
      align
      center/left/right
      alt
      none
      title
      none
      background-color
      each social name has its own default.
      color
      #333333
      border-radius
      3px
      font-family
      Ubuntu, Helvetica, Arial, sans-serif
      font-size
      13px
      href
      Some supported social media networks can use short url
      https://www.facebook.com/sharer/sharer.php?u=[[URL]]
      To keep your href, append -noshare to the name attribute
      e.g. <mj-social-element name="twitter-noshare" href="my-unchanged-url">Twitter</mj-social-element>
      src
      each social name has its own default. Default images are transparent. Use background-color to set icon color
      icon-height
      20px. percent/px
      icon-size
      20px. percent/px
      icon-padding
      0px
      line-height
      22px
      mode
      horizontal (vertical)
      name
      Supported with a share url
      facebook google pinterest linkedin twitter tumblr xing
      Without a share URL
      github instagram web snapchat youtube vimeo medium soundcloud dribble
      padding
      10px 25px
      text-decoration
      none
      text-padding
      4px 4px 4px 0
      target
      _blank

    • Version 3 

      <mj-social color="#333333" align="left" 
text-mode="false" icon-size="16px" 
padding-right="25" padding-bottom="10" padding-top="10" 
mode="horizontal" 
display="linkedin:url facebook:url twitter:url youtubec:url" 
linkedin-href="..."
linkedin-content=" "
linkedin-icon-color="#A7A6A4" 
youtubec-href="..."
youtubec-content=" "
youtubec-icon-color="#A7A6A4" youtubec-icon="path to youtube icon image" />
      mj-social

      attributes

      text-mode
      default true, if true, *-content must be defined
      (no term)
      font-family
      color
      text color, default #333333
      align
      center
      icon-size
      default 20px
      padding
      default 10px 25px
      mode
      defualt horizontal, or vertical
      display
      default facebook twitter google. could be facebook:url, can add custom social network e.g. youtube
      *-href, *-content, *-rel, *-icon, *-icon-color
      * could be facebook, twitter, google, linkedin, pinterest, instagram
      • *-icon for default facebook, twitter, etc. cannot be modified
      • *-icon-color has to be defined for custom social medias e.g. youtubec
  5. mj-divider

    Try to add horizontal border with some spacing using mj-divider

    border-color
    #000000
    border-style
    solid, dashed, dotted
    border-width
    4px
    width
    100% px/percent
    padding
    10px 25px
    container-background-color
    <mj-section padding-bottom="0" background-color="white">
  <mj-column width="100%">
    <mj-image src="https://avatars0.githubusercontent.com/u/16115896?v=3&s=200" width="50px" />
    <mj-divider padding-top="20" padding-bottom="0" padding-left="0" padding-right="0" border-width="1px" border-color="#f8f8f8" />
  </mj-column>
</mj-section>
  6. mj-spacer

    blank space. For some reason, try adding a section with a spacer only so that the email has the content in full width

    height
    20px
    (no term)
    css-class
    <mj-section>
  <mj-column>
    <mj-text>A first line of text</mj-text>
    <mj-spacer height="50px" />
    <mj-text>A second line of text</mj-text>
  </mj-column>
</mj-section>
  7. mj-table
    • only accepts plain html
    • Use mj-table to have fixed width that's not gonna change when in mobile. Or width will be 100%
    • mj-table always, as a whole, align left while mjml:tag:mj-social aligns center. To center, use mj-raw to mjml:raw:center table
    • Attributes
      font-family
      Ubuntu, Helvetica, Arial, sans-serif
      font-size
      13px
      line-height
      22px (%/px)
      padding
      10px 25px
      width
      default 100%. With default left+right padding 25*2, the maximum width can be less than 300px. Otherwise the table has 100% width
      table-layout
      auto (fixed/initial/inherit)

    mjml:tag:mj-table:social Use it to align icons with links. Notice the img has fixed width. mj-table has fixed width including padding 25*5+5*(5-1)=165px. If mj-table has no width, columns will have equal width. 

    <mj-table width="165px">
  <!-- the style in tr is very important for Outlook to have the correct column height-->
  <tr style="list-style: none;line-height:1">
    <td>
  <a href="https://twitter.com/RecastAI">
    <img width="25" src="https://cdn.recast.ai/newsletter/twitter.png" />
  </a>
    </td>
    <td>
  <a href="https://www.facebook.com/recastAI">
    <img width="25" src="https://cdn.recast.ai/newsletter/facebook.png" />
  </a>
    </td>
    <td>
  <a href="https://medium.com/@RecastAI">
    <img width="25" src="https://cdn.recast.ai/newsletter/medium.png" />
  </a>
    </td>
    <td>
  <a href="https://www.youtube.com/channel/UCA0UZlR8crpgwFiVaSTbVWw">
    <img width="25" src="https://cdn.recast.ai/newsletter/youtube.png" />
  </a>
    </td>
    <td>
  <a href="https://plus.google.com/u/0/+RecastAi">
    <img width="25" src="https://cdn.recast.ai/newsletter/google%2B.png" />
  </a>
    </td>
  </tr>
</mj-table>
    • For borders on 4 sides (bottom/top/left/right), I had trouble with setting in <tr>. Define 4 sided borders in <td> is fine.
    <mj-column>
  <mj-table>
    <tr style="border-bottom:1px solid #ecedee;text-align:left;padding:15px 0;">
  <th style="padding: 0 15px 0 0;">Year</th>
  <th style="padding: 0 15px;">Language</th>
  <th style="padding: 0 0 0 15px;">Inspired from</th>
    </tr>
    <tr>
  <td style="padding: 0 15px 0 0;">1995</td>
  <td style="padding: 0 15px;">PHP</td>
  <td style="padding: 0 0 0 15px;">C, Shell Unix</td>
    </tr>
    <tr>
  <td style="padding: 0 15px 0 0;">1995</td>
  <td style="padding: 0 15px;">JavaScript</td>
  <td style="padding: 0 0 0 15px;">Scheme, Self</td>
    </tr>
  </mj-table>
</mj-column>

    <mj-section>
  <mj-column>
    <mj-table>
  <tr style="list-style: none;line-height:1">
    <td width="46"><img width="46" src="http://via.placeholder.com/46x32/" /></td>
    <td width="10">&nbsp;</td>
    <td>
      <br>
      <span style="font-size:18px;font-weight:200;">Contact Us</span><br>
      <a href="tel:18881231234" target="_blank" style="text-decoration:none;color:#545759;">1-888-123-1234</a><br>
      <a href="mailto:a@xyz.com" target="_blank" style="text-decoration:none;color:#545759;">a@xyz.com</a><br>
      <a href="http://a.com" target="_blank" style="text-decoration:none;color:#545759;">www.a.com</a>
    </td>
  </tr>
    </mj-table>
  </mj-column>
</mj-section>
  8. mj-raw
    • Not to be rendered by MJML engine
    • Unclosed HTML tags inside mj-raw will be closed
    • No attributes

    mjml:raw:center table This won't work for Yahoo! Mail but works on all clients 

    <mj-column>
  <mj-text>...</mj-text>
  <!-- each mj-element under mj-column is <tr><td> content </td></tr> -->

  <!-- align center -->
  <mj-raw>
    <tr>
  <td>
    <table width="100%" cellpadding="0" cellspacing="0" border="0" role="presentation" style="vertical-align:top">
      <tr>
        <td align="center" style="font-size:0px;padding:10px 25px;padding-bottom:0px;word-break:break-word;">

      <!-- social icons -->
      <table width="95" cellpadding="0" cellspacing="0" border="0" role="presentation" style="vertical-align:top">
        <tr style="list-style:none;line-height:1;">
          <td>
            <a href="#" target="_blank">
          <img width="20" height="20" src="#" alt="" />
            </a>
          </td>
          <td>
            <a href="#" target="_blank">
          <img width="20" height="20" src="#" alt="" />
            </a>
          </td>
          <td>
            <a href="#" target="_blank">
          <img width="20" height="20" src="#" alt="" />
            </a>
          </td>
          <td>
            <a href="#" target="_blank">
          <img width="20" height="20" src="#" alt="" />
            </a>
          </td>
        </tr>
      </table>
      <!-- social icons. -->

        </td>
      </tr>
    </table>
  </td>
    </tr>
  </mj-raw>
  <!-- align center. -->

  <!-- align right, pay attention to padding-right 25px -->
  <mj-raw>
    <tr>
  <td>
    <table width="100%" cellpadding="0" cellspacing="0" border="0" role="presentation" style="vertical-align:top">
      <tr>
        <td align="right" style="font-size:0px;padding:10px 25px;padding-bottom:0px;word-break:break-word;">

      <!-- social icons -->
      <table width="95" cellpadding="0" cellspacing="0" border="0" role="presentation" style="vertical-align:top">
        <tr style="list-style:none;line-height:1;">
          <td>
            <a href="#" target="_blank">
          <img width="20" height="20" src="#" alt="" />
            </a>
          </td>
          <td>
            <a href="#" target="_blank">
          <img width="20" height="20" src="#" alt="" />
            </a>
          </td>
          <td>
            <a href="#" target="_blank">
          <img width="20" height="20" src="#" alt="" />
            </a>
          </td>
          <td>
            <a href="#" target="_blank">
          <img width="20" height="20" src="#" alt="" />
            </a>
          </td>
        </tr>
      </table>
      <!-- social icons. -->

      <!-- mj-image with href HTML rendered result -->
      <table cellpadding="0" cellspacing="0" border="0" role="presentation" style="border-collapse:collapse;border-spacing:0px;">
        <tr>
          <td style="width:110px;">
            <a href="#" target="_blank">
          <img width="110" height="auto" src="#" alt="" style="border:0;display:block;outline:none;text-decoration:none;height:auto;width:100%;" />
            </a>
          </td>
        </tr>
      </table>
      <!-- mj-image with href HTML rendered result. -->

        </td>
      </tr>
    </table>
  </td>
    </tr>
  </mj-raw>
  <!-- align right, pay attention to padding-right 25px. -->

</mj-column>
    1. Old align center
      <mj-style inline="inline">
  .tbl-cell-1 {
  font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;
  font-size:12px;
  padding:0px 0px 10px 0px;

  width:30px;
  }
  .tbl-cell-2, .tbl-cell-3 {
  width:30px;
  }
</mj-style>

<mj-column>
  <mj-text>...</mj-text>
  <!-- each mj-element under mj-column is <tr><td> content </td></tr> -->
  <!-- OLD align center -->
  <mj-raw>
    <tr>
  <td>
    <table align="center" width="110" cellpadding="0" cellspacing="0" border="0">
      <tr>
        <td align="center" class="tbl-cell-1"><a href="#">
        <img width="30" src="twitter.png" />
        </a></td>
        <td align="center" class="tbl-cell-2"><a href="#">
        <img width="30" src="twitter.png" />
        </a></td>
        <td align="center" class="tbl-cell-3"><a href="#">
        <img width="30" src="twitter.png" />
        </a></td>
      </tr>
    </table>
  </td>
    </tr>
  </mj-raw>
  <!-- OLD align center. -->
</mj-column>
  9. mj-hero

    Display a section with a background image and some content inside (mj-text, mj-button, mj-image …) wrapped in mj-hero-content component

    mjml:tag:mj-hero

  10. mj-invoice
  11. mj-location
  12. mj-navbar

    https://mjml.io/documentation/#mjml-navbar So far, all have been tested only show expanded items.

  13. mj-carousel

    https://mjml.io/documentation/#mjml-carousel Outlook and OWA only shows the first image. Works for Gmail.com and Android Gmail.

  14. mj-accordion

    https://mjml.io/documentation/#mjml-accordion Every attribute in `mj-accordion` are applied to `mj-accordion-element` unless you overide them on `mj-accordion-element`

    So far, all have been tested only show expanded items.

    The "hamburger" feature only work on mobile device with all iOS mail client, for others mail clients the render is performed on an normal way, the links are displayed inline and the hamburger is not visible. 

    <mjml>
  <mj-head>
    <mj-attributes>
  <mj-accordion border="none" padding="1px" />
  <mj-accordion-element icon-wrapped-url="http://i.imgur.com/Xvw0vjq.png" icon-unwrapped-url="http://i.imgur.com/KKHenWa.png" icon-height="24px" icon-width="24px" />
  <mj-accordion-title font-family="Roboto, Open Sans, Helvetica, Arial, sans-serif" background-color="#fff" color="#031017" padding="15px" font-size="18px" />
  <mj-accordion-text font-family="Open Sans, Helvetica, Arial, sans-serif" background-color="#fafafa" padding="15px" color="#505050" font-size="14px" />
    </mj-attributes>
  </mj-head>

  <mj-body>
    <mj-section padding="20px" background-color="#ffffff">
  <mj-column background-color="#dededd">
    <mj-accordion>
      <mj-accordion-element>
        <mj-accordion-title>Why use an accordion?</mj-accordion-title>
        <mj-accordion-text>
      <span style="line-height:20px">
        Because emails with a lot of content are most of the time a very bad experience on mobile, mj-accordion comes handy when you want to deliver a lot of information in a concise way.
      </span>
        </mj-accordion-text>
      </mj-accordion-element>
      <mj-accordion-element>
        <mj-accordion-title>How it works</mj-accordion-title>
        <mj-accordion-text>
      <span style="line-height:20px">
        Content is stacked into tabs and users can expand them at will. If responsive styles are not supported (mostly on desktop clients), tabs are then expanded and your content is readable at once.
      </span>
        </mj-accordion-text>
      </mj-accordion-element>
    </mj-accordion>
  </mj-column>
    </mj-section>
  </mj-body>
</mjml>
7.10.12.17. mj-group to group columns

Prevent columns from stacking on mobile.

Columns inside a group must have a width in percentage or auto sizing, but not in pixel

Section can have both mj-column and mj-group.

iOS 9 :: always remove space between columns inside a mj-group. Or export HTML using minify.

width
100 / number of children in section (%/px)
vertical-align
top
background-color
default n/a
7.10.12.18. Background Image

Docs from Litmus

background-url can only be used in mj-hero, mj-section, mj-wrapper background-position can only be set in mj-hero and top center for other mjml elements

7.10.12.19. Change default padding

Use this to change left/right padding in % is not consistent especially in Outlook 2003-2016. For Outlook desktop, you may need to hard set mj-image width. 

<mjml>
  <mj-head>
    <mj-attributes>
      <mj-all padding-left="0px"></mj-all>
      <mj-all padding-right="0px"></mj-all>
    </mj-attributes>
  </mj-head>

<mj-section padding-left="12%" padding-right="12%">
</mj-section>
7.10.12.20. Caveats
  • mj-image inside mj-column will always have a maximum fixed width
    • If there's one mj-column, mj-image is 100% of the mj-column
    • If there're 2 mj-column, mj-image is 50% of the whole width = 300px
    • If there're 3 mj-column, mj-image is 196px
    • You can align mj-image left or right inside a mj-column. And when you do that, pay attention to the padding. Usually don't set padding:0

    • When in mobile, <p> has padding-left:10px. So you could do 

        <mj-section full-width="full-width">
<mj-group>
  <mj-column width="50%">
    <mj-image src="logo1.jpg" width="103" align="left" padding="10px 0 0 10px" />
  </mj-column>
  <mj-column width="50%">
    <mj-image src="logo2.jpg" width="96" align="right" padding="10px 10px 0 0"/>
  </mj-column>
</mj-group>
  </mj-section>

mj-section, mj-column don't have left and right padding by default.

mj-image has padding:10px 25px by default

7.10.13. Non breaking css:content

7.10.14. Remove spaces between HTML end tags and start tags email:spaces between html tags

// m flag for multiple lines
// IE start and end if statements need a speparate line
// <tr><td>
// <!--[if mso | IE]><table><tr><td>
// <!--<![endif]-->
$mini_code = preg_replace('~>\s+<(?!\!)~m','><',$code);

7.10.15. Don't use anchor link!

7.11. Phone number

<a href="tel:+18475555555">1-847-555-5555</a>

// javascript window.open('tel:+11231231234');

7.12. Link html:link

  • Also can be set in HTML response header as Link. Refer to header:link

7.12.1. media="print" html:link:media

Refer to css:@media
<link rel="stylesheet" href="print.css" media="print">

7.12.2. rel="preload" html:link:rel:preload

  • Not supported in IE11-, Edge, Safari 11-, iOS Safari 11.2-
    • Unsupported browsers will just ignore
  • Resources loaded via <link rel="preload"> are stored locally in the browser, and are effectively inert until they're referenced in the DOM, JavaScript, or CSS
  • Does not block window.onload event
  • To load late-discovered critical resources which will be loaded very soon at current request
  • Compliant with Content-Security-Policy and http header Accept

  • Preload script 

    <link rel="preload" href="used-later.js" as="script" type="text/javascript" />

<!-- json -->
<link rel="preload" href="/graphql/query?id=12345" as="fetch" type="application/json" />

<!-- ...other HTML... -->
<script>
  // Later on, after some condition has been met, we run the preloaded
  // JavaScript by inserting a <script> tag into the DOM.
  var usedLaterScript = document.createElement('script');
  usedLaterScript.src = 'used-later.js';    
  document.body.appendChild(usedLaterScript)
</script>
  • <link rel="preload" as="image" href="logo.jpg"/>
  • html:link:rel:preload:font
    • <link rel="preload" href="font.woff2" as="font" type="font/woff2" crossorigin>
    • Font needs crossorigin even it's on the same origin. type is also needed
    • All browsers that support preloading also support WOFF2, so font/woff2 is the format that you should preload
    • preload won't have the ability to css:font-face:local, so the font will be always downloaded
  • html:video:preload

  • If you don't want to preload a resource at page load but want to preload it before it's used 

    var preload = document.createElement("link");
link.href = "myscript.js";
link.rel = "preload";
link.as = "script";
document.head.appendChild(link);

// later
var script = document.createElement("script");
script.src = "myscript.js";
document.body.appendChild(script);

  • do something when resource is downloaded which is different from window.onload 

    <link rel="preload" as="style" href="async_style.css" onload="this.rel='stylesheet'">

<script>
var asyncScriptOnLoad = function() {
  var script = document.createElement('script');
  script.src = this.href;
  document.body.appendChild(script);
};
</script>

<link rel="preload" as="script" href="async_script.js"
onload="asyncScriptOnLoad();">

  • Detect if <link rel="preload"> is supported 

    const preloadSupported = () => {
  const link = document.createElement('link');
  const relList = link.relList;
  if (!relList || !relList.supports)
return false;
  return relList.supports('preload');
};
  • HTTP/2 Server Push can push resources to browser before browser sends the request for them. Which means Push can send resources before HTML even started to be sent to the browser. Resources have to be on the same origin

7.12.3. rel="prefetch"

  • Has better browser support than rel="preload"
  • Download non-critical resources earlier
  • Prefetch non-recursively fetches the top level resource not the child resources

  • Downloads the same css 2 times 

    <link rel="prefetch" href="optional.css">
<link rel="stylesheet" href="optional.css">

7.12.4. rel="dns-prefetch"

  • rel="preconnect" has less browser support and it's a superset of dns-prefetch 

    <link rel="dns-prefetch" href="http://www.spreadfirefox.com/">
<link rel="dns-prefetch" href="//www.spreadfirefox.com">

7.12.5. preload vs prefetch and Network Prioritisation

Network Prioritisation
https://medium.com/reloading/preload-prefetch-and-priorities-in-chrome-776165961bbf
(no term)
Priority shown in DevTools:
Highest
layout-blocking
CSS inside <head>
match
(no term)
Font
(no term)
XHR (sync)
(no term)
<link rel=preload as=style>
High
load in layout-blocking phase
<script> inside <head>
requested before any non-preloaded images
(no term)
<link rel=preload as=script>
(no term)
<link rel=preload as=font>
(no term)
Import
(no term)
Image (in viewport)
(no term)
<link rel=preload> with no as will behave as XHR
(no term)
XSL
(no term)
XHR/fetch* (async)
Medium
Load one-at-a-time in layout-blocking phase
<script>
requested after any non-preloaded images
(no term)
Favicon
Low
Load one-at-a-time in layout-blocking phase
  • <script async>
  • <script defer>
  • Injected <script>
  • Image
  • <link rel=preload as="image">
  • Media
  • SVG
Lowest
Load one-at-a-time for layout-blocking phase. after current page is done loading and there's bandwidth available
CSS
mismatch
(no term)
<link rel="prefetch">

7.12.6. rel="canonical" and rel="alternate" html:link:rel:canonical html:link:rel:alternate

  • A canonical URL is the URL of the page that Google thinks is most representative from a set of duplicate pages
  • Different language versions of a single page are considered duplicates only if the main content is in the same language (that is, if only the header, footer, and other non-critical text is translated, but the body remains the same, then the pages are considered to be duplicates). In general, a translated page is not a duplicate
  • Use absolute paths
  • 301 redirect is the best way to ensure other duplicate pages are removed and directed to the canonical URL
<!-- desktop page: http://www.example.com/page-1 -->
<link rel="alternate" media="only screen and (max-width: 640px)" href="http://m.example.com/page-1">

<!-- mobile page: http://m.example.com/page-1 -->
<link rel="canonical" href="http://www.example.com/page-1">

<!-- sitemap -->
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
    xmlns:xhtml="http://www.w3.org/1999/xhtml">
    <url>
    <loc>http://www.example.com/page-1/</loc>
    <xhtml:link
        rel="alternate"
        media="only screen and (max-width: 640px)"
        href="http://m.example.com/page-1" />
    </url>
</urlset>

7.12.7. rel="manifest.json" html:link:rel:manifest

<link rel="manifest" href="/manifest.json"> 

{
  "short_name": "Maps", // 12 characters
  "name": "Google Maps", // your app name, 45 characters

  // If you want to have your own size, provide icons in increments of 48px or 128px.
  "icons": [

    {
  "src": "/images/icons-192.png",
  "type": "image/png",
  "sizes": "192x192"
    },
    {
  "src": "/images/icons-512.png",
  "type": "image/png",
  "sizes": "512x512"
    }
  ],

  "start_url": "/?utm_source=pwa", // the page the app should start with. Tracking query string can be added e.g. track how often the app is launched.
  "background_color": "#3367D6",

  "display": "standalone",
  // standalone: separte from the browser and runs in its own window.
  // fullscreen: no browser UI
  // minimal-ui: not supported by Chrome. Similar to fullscreen but might have other navigation (back, forward, reload)
  // browser: a standard browser experience

  // "orientation": "landscape",
  // Enforce an orientation

  "scope": "/maps/",
  // if current page is not inside this directory, then the user exists the app
  // If scope is not defined, then default is the directory that the app manifest is served from
  // can be relative path e.g. ../
  // start_url must be in the scope
  // start _url is relative to the path defined in scope
  // start_url starting with / will always be the root of the origin

  "theme_color": "#3367D6"
}

7.12.8. rel="icon" rel="apple-touch-startup-image" rel="apple-touch-icon"

Refer to html:address bar https://developers.google.com/web/fundamentals/design-and-ux/browser-customization/

Chrome and Opera in scale of 48px (muliple sizes can be added) 

<link rel="icon" sizes="192x192" href="icon.png">

IE Tiles :: 4 sizes. Listed are standard size. 1.8 times is better. 

<meta name="application-name" content="CenturyCutCook" />
<meta name="msapplication-TitleColor" content="#009900" />

<meta name="msapplication-sqaure70x70logo" content="icon_smalltitle.png">
<meta name="msapplication-sqaure150x150logo" content="icon_mediumitle.png">
<meta name="msapplication-sqaure310x150logo" content="icon_widetitle.png">
<meta name="msapplication-sqaure310x310logo" content="icon_largetitle.png">

Apple Icon :: Default 60x60 

<link rel="apple-touch-icon" href="touch-icon-iphone.png">
<link rel="apple-touch-icon" sizes="76x76" href="touch-icon-ipad.png">
<link rel="apple-touch-icon" sizes="120x120" href="touch-icon-iphone-retina.png">
<link rel="apple-touch-icon" sizes="152x152" href="touch-icon-ipad-retina.png">

Apple :: Startup Image. By default, Safari shows a blank screen during load time and after multiple loads a screenshot of the previous state of the app. Base Size :: 320x480. But different devices ask for different sizes. Follow this to implement 

<link rel="apple-touch-startup-image" href="/startup.png">

<!-- iPad retina portrait startup image -->
<link href="https://placehold.it/1536x2008"
      media="(device-width: 768px) and (device-height: 1024px)
                 and (-webkit-device-pixel-ratio: 2)
                 and (orientation: portrait)"
      rel="apple-touch-startup-image">

<!-- iPad retina landscape startup image -->
<link href="https://placehold.it/1496x2048"
      media="(device-width: 768px) and (device-height: 1024px)
                 and (-webkit-device-pixel-ratio: 2)
                 and (orientation: landscape)"
      rel="apple-touch-startup-image">

<!-- iPad non-retina portrait startup image -->
<link href="https://placehold.it/768x1004"
      media="(device-width: 768px) and (device-height: 1024px)
                 and (-webkit-device-pixel-ratio: 1)
                 and (orientation: portrait)"
      rel="apple-touch-startup-image">

<!-- iPad non-retina landscape startup image -->
<link href="https://placehold.it/748x1024"
      media="(device-width: 768px) and (device-height: 1024px)
                 and (-webkit-device-pixel-ratio: 1)
                 and (orientation: landscape)"
      rel="apple-touch-startup-image">

<!-- iPhone 6 Plus portrait startup image -->
<link href="https://placehold.it/1242x2148"
      media="(device-width: 414px) and (device-height: 736px)
                 and (-webkit-device-pixel-ratio: 3)
                 and (orientation: portrait)"
      rel="apple-touch-startup-image">

<!-- iPhone 6 Plus landscape startup image -->
<link href="https://placehold.it/1182x2208"
      media="(device-width: 414px) and (device-height: 736px)
                 and (-webkit-device-pixel-ratio: 3)
                 and (orientation: landscape)"
      rel="apple-touch-startup-image">

<!-- iPhone 6 startup image -->
<link href="https://placehold.it/750x1294"
      media="(device-width: 375px) and (device-height: 667px)
                 and (-webkit-device-pixel-ratio: 2)"
      rel="apple-touch-startup-image">

<!-- iPhone 5 startup image -->
<link href="https://placehold.it/640x1096"
      media="(device-width: 320px) and (device-height: 568px)
                 and (-webkit-device-pixel-ratio: 2)"
      rel="apple-touch-startup-image">

<!-- iPhone < 5 retina startup image -->
<link href="https://placehold.it/640x920"
      media="(device-width: 320px) and (device-height: 480px)
                 and (-webkit-device-pixel-ratio: 2)"
      rel="apple-touch-startup-image">

<!-- iPhone < 5 non-retina startup image -->
<link href="https://placehold.it/320x460"
      media="(device-width: 320px) and (device-height: 480px)
                 and (-webkit-device-pixel-ratio: 1)"
      rel="apple-touch-startup-image">

7.12.9. rel="noopener" html:link:rel:noopener

  • HTML link to another domain with target="_blank", better add this so that window.opener object doesn't exist on the new page in new tab 

    <a href="anotherdomain.gg" target="_blank" rel="noopener">...</a>

7.13. Autofill, Input Types

https://developers.google.com/web/updates/2015/06/checkout-faster-with-autofill

Autofill hint set :: the address or contact info type

autocomplete="shipping locality"
"shipping" or "billing" meaning the field is part of the shipping address or contact information.

These apply to telephone, emails and instant messaging

  • home, work, mobile, fax, pager

section-* may be added with autofill hint set. The whole thing is called autofill scope.

Autofill fields :: have to match <input> type e.g. text, select, etc.

Wrap <input> inside <form> in order for autocomplete to work in Chrome.

[section-](optional) [shipping|billing](optional) [home|work|mobile|fax|pager](optional) [autofill field name] 

<label for="frmNameCC">Name on card</label>
<input name="ccname" id="frmNameCC" required placeholder="Full Name" autocomplete="cc-name">

<label for="frmCCNum">Card Number</label>
<input name="cardnumber" id="frmCCNum" required autocomplete="cc-number">

<label for="frmCCCVC">CVC</label>
<input name="cvc" id="frmCCCVC" required autocomplete="cc-csc">

<label for="frmCCExp">Expiry</label>
<input name="cc-exp" id="frmCCExp" required placeholder="MM-YYYY" autocomplete="cc-exp">

<label for="frmCityS">City</label>
<input name="ship-city" required id="frmCityS" placeholder="New York" autocomplete="shipping locality">

<fieldset>
 <legend>Ship the blue gift to...</legend>
 <p> <label> Address:     <textarea name=ba autocomplete="section-blue shipping street-address"></textarea> </label>
 <p> <label> City:        <input name=bc autocomplete="section-blue shipping address-level2"> </label>
 <p> <label> Postal Code: <input name=bp autocomplete="section-blue shipping postal-code"> </label>
</fieldset>
<fieldset>
 <legend>Ship the red gift to...</legend>
 <p> <label> Address:     <textarea name=ra autocomplete="section-red shipping street-address"></textarea> </label>
 <p> <label> City:        <input name=rc autocomplete="section-red shipping address-level2"> </label>
 <p> <label> Postal Code: <input name=rp autocomplete="section-red shipping postal-code"> </label>
</fieldset>

7.13.1. Input Types

Input types

url
validate http://, ftp:// or mailto:
tel
don't validate
email
may validate? May use attribute multiple
(no term)
search
number
iOS requires pattern="\d*" to show the numeric keyboard
(no term)
range
(no term)
datetime-local
(no term)
date
(no term)
time
(no term)
month
(no term)
color

Provide suggestions for text field 

<label for="frmFavChocolate">Favorite Type of Chocolate</label>
<input type="text" name="fav-choc" id="frmFavChocolate" list="chocType">
<datalist id="chocType">
  <option value="white">
  <option value="milk">
  <option value="dark">
</datalist>

Validate form :: Constraint Validation API https://developers.google.com/web/fundamentals/design-and-ux/input/forms/

7.14. Special Characters css:content

  • In CSS content attribute, you need to use CSS value
    Convert tool
    http://www.evotech.net/articles/testjsentities.html
    Decimal value
    &#9660;
    Hex value
    &#x25BC;
    CSS Value (Hex)
    \25BC li:before {content:'\25BC';}
    JS Value (Hex)
    \u25BC alert('\u25BC is HTML entity &#9660;')
  • Numeric character reference
    &#nnnn;

    Unicode code point decimal form

    • Can be more than 4 digits
    • ASCII
    &#xhhhh;
    Unicode code point hexadecimal form
    x
    must be lowercase
    h
    mix case but uppercase is usual
  • DTD Standards

    • Usually are predefined in markup languages using DTD

      • HTML5 does not allow to extend or add more character entity references
      • XML now is in XSD (XML Schema) but XSD also follows part of DTD. Can add more character entity references
      • XML has 5 predefined entities, which usually need to be escaped inside an XML tag (as attribute or content)
        &
        &amp;
        <
        &lt;
        >
        &gt;
        '
        &apos;
        "
        &quot;
    • e.g. &nbsp; is a space. Case-sensitive
    • https://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references
  • Unicode
  • &#10003;
    Checkmark heavy
    &#10004;
    Checkmark light
    &#128504;
    Checkmark with square
    &#9745;
  • &nbsp;
  • &#x2011;
  • Quotation mark
    Left Double Quotation Mark “
    &#8220;
    Right Double Quotation Mark ”
    &#8221;
    Left Single Quotation Mark ‘
    &#8216;
    Right Single Quotation Mark (including English possessives and contractions) ’
    &#8217;
    Closing/Right angle quotation mark »
    css is \00BB &#187;
  • &middot; &#183; &#xb7; CSS is \00B7
  • https://www.alt-codes.net/triangle-symbols
    ▼
    &#9660; \25BC \u25BC
    ▾
    &#9662; \25BE \u25BE

7.15. Geolocation

Chrome v50+ only supports geolocation for https

7.15.1. .getCurrentPosition

<button onclick="getLocation()">Get Location</button>
<p id="demo"></p>
<script>
var x = document.getElementById("demo");
function getLocation() {
  if (navigator.geolocation) {
    navigator.geolocation.getCurrentPosition(showPosition, showError);
  }
  else {
    x.innerHTML = "Not supported";
  }
}

function showPosition(position) {
 x.innerHTML = "Latitiude" + position.coords.latitude +
               "Longitude: " + position.coords.longitude;
}

function showError(e) {
  switch (e.code) {
    case: e.PERMISSION_DENIED:
     break;
    case: e.POSITION_UNAVAILABLE;
    case: e.TIMEOUT:
    case: e.UNKNOWN_ERROR:
  }
}
</script>

7.15.2. watchPosition

Same as getCurrentPosition but continuous track.

7.16. Storage

if (typeof(Storage) !== "undefined") {
  // Use localStorage or sessionStorage
  // Store. value is always string
  localStorage.setItem("key","value");
  console.log(localStorage.getItem("key"));
  localStorage.clickcount = Number(localStorage.clickcount) + 1;
  console.log(localStorage.key);
} 
else {
  // Not supported
}

7.17. Application Cache

<html manifest="demo.appcache" .appcache file should be served in text/cache-manifest media type 

CACHE MANIFEST
# Above should be first line
# Files that should be cached
/theme.css
NETWORK
# Files will not be cached
login.asp
# An asterisk indicates all other files require an internet connection
*
FALLBACK
/html/ /offline.html
# offline.html should be served in place of all files in /html/ directory
# if internet is not available

# Change the manifest file to update cache

7.18. ARIA html:ARIA

https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/ARIA_Techniques

7.18.1. role attribute

separator
BS4 .dropdown-divider
group
BS4 .btn-group
button
<a> without href
search
<form> define purpose
presentation
element does not need to be mapped to the accessibility API

7.18.2. aria-* attributes

aria-label
Label or name is used interchangably. e.g. Inside <button> where label text can't be displayed
aria-labelledby
Labelled by other element(s). Refer to other elements using their ids.
  • Can be nested aria-labelledby="parentID child1ID"
  • Say parentID element contains text Men's Shirts and childID element contains text Shop Now, then the resulted name of the element used aria-labelledby has a name "Men's Shirts Shop Now"
  • it can even refer to itself
(no term)

aria-hidden="true" 

<h1 aria-label="Hello World">
  <div class="grid" aria-hidden="true">
<!-- the following are not incluced in any accessibility device -->
<span>H</span>
<span>e</span>
<span>l</span>
<span>l</span>
<span>o</span>

<span>w</span>
<span>o</span>
<span>r</span>
<span>l</span>
<span>d</span>
  </div>
</h1>

7.19. Display price

<span class="price-49-cents">4.9</span>

<style>
  .price-49-cents {
  font-size:70px;
  }
  .price-49-cents:after {
  content; "\00A2 \A per kWh";
  white-space: pre; /* wrap when line break is encountered */
  font-size:16px;
  /* line-height:16px; probably don't need it */
  vertical-align:top;
  display:inline-block;
  }
</style>

7.21. AMP google:amp

7.21.1. Search Engine Discovery

  • https://www.ampproject.org/docs/fundamentals/spec
  • <link rel="amphtml" href="https://www.example.com/url/to/amp/document.html">
  • <link rel="canonical" href="https://www.example.com/url/to/full/document.html">
    If there's only one page and the page is AMP
    <link rel="canonical" href="https://www.example.com/url/to/amp/document.html">

7.21.2. AMP JS Library

<script async src="https://cdn.ampproject.org/v0.js"></script>

7.21.3. Boilerplate

<style amp-boilerplate>body{-webkit-animation:-amp-start 8s steps(1,end) 0s 1 normal both;-moz-animation:-amp-start 8s steps(1,end) 0s 1 normal both;-ms-animation:-amp-start 8s steps(1,end) 0s 1 normal both;animation:-amp-start 8s steps(1,end) 0s 1 normal both}@-webkit-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-moz-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-ms-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-o-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}</style><noscript><style amp-boilerplate>body{-webkit-animation:none;-moz-animation:none;-ms-animation:none;animation:none}</style></noscript>

7.21.4. amp-custom amp:amp-custom

<style amp-custom>
  /* any custom style goes here */
  body {
    background-color: white;
  }
  amp-img {
    background-color: gray;
    border: 1px solid black;
  }
</style>

7.21.5. amp-image amp:amp-image

  • Basics
    • Built-in element, no need to import another AMP runtime
<amp-img src="/static/samples/img/amp.jpg"
  alt="AMP"
  width="475"
  height="268"
  layout="responsive">
  <noscript>
    <img src="/static/samples/img/amp.jpg" width="475" height="268" alt="AMP">
  </noscript>
</amp-img>

<div>
  <amp-img src="https://unsplash.it/300/200/?image=100"
    width="300"
    height="200"
    [src]="myImageUrl">
  </amp-img>
  <button on="tap:AMP.setState({ myImageUrl: 'https://unsplash.it/300/200/?image=200' })">Change image</button>
</div>

7.21.6. amp-bind amp-bind-macro amp:amp-bind

7.21.6.1. amp-bind
<script async custom-element="amp-bind" src="https://cdn.ampproject.org/v0/amp-bind-0.1.js"></script>

<div>
  <div hidden
   [hidden]="hideGreeting">Hello World</div>
  <button on="tap:AMP.setState({ hideGreeting: false })">Show greeting</button>
</div>

<div>
  <div>Hello <span [text]="myText">World</span></div>
  <button on="tap:AMP.setState({ myText: 'AMP' })">Change text</button>
</div>

<div>
  <div class="background-red"
   [class]="myClass">Hello World</div>
  <button on="tap:AMP.setState({ myClass: 'background-green' })">Change class</button>
</div>

<div>
  <amp-img src="https://unsplash.it/400/200"
       width="200"
       [width]="myImageDimension.width"
       height="100"
       [height]="myImageDimension.height">
  </amp-img>
  <button on="tap:AMP.setState({
      myImageDimension: {
      width: 400,
      height: 200
      }
      })">
    Change size
  </button>
</div>

<div>
  <select on="change:AMP.setState({ option: event.value })">
    <option value="0">No selection</option>
    <option value="1">Option 1</option>
    <option value="2">Option 2</option>
  </select>
  <div hidden
   [hidden]="option != 1">
    Option 1
  </div>
  <div hidden
   [hidden]="option != 2">
    Option 2
  </div>
</div>

<!-- initialize state -->
<div>
  <amp-state id="myText">
    <script type="application/json">
  "World"
    </script>
  </amp-state>
  <div>1. Hello <span [text]="undefinedText">World</span></div>
  <div>2. Hello <span [text]="myText">World</span></div>
  <button on="tap:AMP.setState({ myText: 'AMP' })">Change state</button>
</div>

<!-- remote state -->
<!-- empty AMP.setState() triggers all amp-bind's to evaluate -->
<div>
  <amp-state id="myRemoteState"
         src="/static/samples/json/websites.json"></amp-state>
  <amp-list layout="fixed-height"
        height="0"
        [height]="18 * myRemoteState.items.length"
        [src]="myRemoteState.items"
        binding="no">
    <template type="amp-mustache">
  <div><a href="{{url}}">{{title}}</a></div>
    </template>
  </amp-list>
  <button on="tap:AMP.setState({})">Show websites</button>
</div>

<!-- refresh one state -->
<div>
  <amp-state id="currentTime"
         src="/documentation/examples/api/time"></amp-state>
  <button on="tap:currentTime.refresh">
    Refresh
  </button>
  <div [text]="currentTime.time"></div>
</div>

<!-- push state -->
<!-- browsser's back button -->
<div>
  <amp-state id="count">
    <script type="application/json">
  1
    </script>
  </amp-state>
  <div>Item <span [text]="count">1</span></div>
  <button on="tap:AMP.pushState({ count: count + 1 })">Increase count</button>
</div>

<!-- debounce input events -->
<div>
  <amp-state id="name">
    <script type="application/json">
  "?"
    </script>
  </amp-state>
  <input id="name-input"
    placeholder="Enter a name"
    on="input-throttled:AMP.setState({ name: event.value })">
  <div>Hello <span [text]="name">?</span></div>
</div>
7.21.6.2. amp-bind-macro
<div>
  <amp-bind-macro id="circleArea"
          arguments="radius"
          expression="3.14 * radius * radius" />
  <input type="number"
     min="0"
     max="100"
     value="0"
     on="input-throttled:AMP.setState({ radius: event.value })">
  <div>
    The circle has an area of <span [text]="circleArea(radius)">0</span>.
  </div>
</div>

7.21.7. amp-carousel amp:amp-carousel

  • Basics 

    <script async custom-element="amp-carousel" src="https://cdn.ampproject.org/v0/amp-carousel-0.1.js"></script>

<amp-carousel height="300"
  layout="fixed-height"
  type="carousel">
  <amp-img src="/static/samples/img/image1.jpg"
width="400"
height="300"
alt="a sample image"></amp-img>
  <amp-img src="/static/samples/img/image2.jpg"
width="400"
height="300"
alt="another sample image"></amp-img>
  <amp-img src="/static/samples/img/image3.jpg"
width="400"
height="300"
alt="and another sample image"></amp-img>
</amp-carousel>
  • https://amp.dev/documentation/examples/multimedia-animations/image_galleries_with_amp-carousel/
  • Attributes
    type
    • carousel
    • may use autoplay at d:5000 milliseconds, use delay to override
    • Style buttons by targeting .amp-carousel-button-prev and .amp-carousel-button-next

7.21.8. amp-video amp:amp-video

  • Basics 

    <script async custom-element="amp-video" src="https://cdn.ampproject.org/v0/amp-video-0.1.js"></script>

<amp-video width="480"
  height="270"
  src="/static/samples/video/tokyo.mp4"
  poster="/static/samples/img/tokyo.jpg"
  layout="responsive"
  controls>
  <div fallback>
<p>Your browser doesn't support HTML5 video.</p>
  </div>
  <source type="video/mp4"
src="/static/samples/video/tokyo.mp4">
  <source type="video/webm"
src="/static/samples/video/tokyo.webm">
</amp-video>

7.21.9. amp-position-observer amp:amp-position-observer

  • https://amp.dev/documentation/components/amp-position-observer/
  • Use it with amp:amp-animation and several video players are the only components that allow low trust events to trigger their actions. Using it alone does not do anything
  • It dispatches events
    enter
    can be tied to start action of amp:amp-animation
    exit
    can be tied to pause action of amp:amp-animation
    scroll:<Position in Viewport As a Percentage>
    can be tied to seekTo action of amp:amp-animation
  • Attributes
    target
    ID of the element to observe. If not specified, the parent of this tag is used as the target
    intersection-ratios
    d:0 between 0 and 1. How much of the target should be visible in the viewport before amp-position-observer triggers any events
    0
    trigger enter when a single pixel of the target comes into vp and trigger exit when the very last pixel goes out of the vp
    0.5
    trigger enter as soon as 50% of the target comes into vp and trigger exit when less than 50% is in the vp
    1
    trigger enter when fully visible and trigger event when a single pixel goes out of vp
    0 1
    trigger enter when top=0 visible and trigger event when bottom=1
    viewport-margins
    100px
    shrink the vp by 100px from the top and 100px from the bottom
    25vh
    shring the vp by 25% from the top and 25% from the bottom. Only consider the middle 50% of the viewport
    100px 10vh
    shrink the vp by 100px from the top and 10% from the bottom
    once
    only trigger the enter and exit once
<script async custom-element="amp-position-observer" src="https://cdn.ampproject.org/v0/amp-position-observer-0.1.js"></script>

7.21.10. amp-animation amp:amp-animation

<script async custom-element="amp-animation" src="https://cdn.ampproject.org/v0/amp-animation-0.1.js"></script>
<script async custom-element="amp-position-observer" src="https://cdn.ampproject.org/v0/amp-position-observer-0.1.js"></script>

<div class="parallax-image-window">
    <amp-position-observer on="scroll:parallaxTransition.seekTo(percent=event.percent)"
               intersection-ratios="0"
               viewport-margins="15vh"
               layout="nodisplay">
    </amp-position-observer>

    <a href="%%CLICK_URL_UNESC%%%%DEST_URL%%" target="_blank">
    <amp-img id="parallaxImage"
         width="300"
         height="600"
         layout="responsive"
         src="%%VIEW_URL_UNESC%%%%FILE:JPG1%%"
         alt="Picture of an elephant"></amp-img>
    </a>
</div>

<amp-animation id="parallaxTransition"
       layout="nodisplay">
    <script type="application/json">
    {
        "duration": "1",
        "fill": "both",
        "direction": "normal",
        "animations": [{
        "selector": "#parallaxImage",
        "keyframes": [
            {"transform": "translateY(-600px)"}
        ]
        }]
    }
    </script>
</amp-animation>

7.21.11. Actions and Events amp:event

Basics

https://amp.dev/documentation/guides-and-tutorials/learn/amp-actions-and-events/ 

<input id="name-input"
   placeholder="Enter a name"
   on="input-throttled:AMP.setState({ name: event.value })">
Syntax
eventName:targetId[.methodName[(arg1=value, arg2=value)]]
  • eventName
    multiple events
    on="submit-success:lightbox1;submit-error:lightbox2"
    multiple events for one action
    on="tap:target1.actionA,target2.actionB"
  • refer to amp:event:target
  • is also called action
  • could be
    unquoted
    simple-value
    quoted
    "string value" or 'string value'
    bool
    true false
    numbers
    11 1.1
    event data
    event.someDataVariableName
(no term)
all elements events
  • tap
  • hide
  • show
  • toggleVisibility
  • toggleClass(class=STRING, force=BOOLEAN)
  • scrollTo(duration=INTEGER, position=STRING)
  • focus
(no term)
input events
input-throttled
same as change but at most once every 100ms

7.21.12. Layout & media queries amp:layout

attribute layout for all elements
https://amp.dev/documentation/guides-and-tutorials/develop/style_and_layout/control_layout/
  • nodisplay

7.21.13. amp-analytics amp:amp-analytics

https://amp.dev/documentation/components/amp-analytics/

  • <script async custom-element="amp-analytics" src="https://cdn.ampproject.org/v0/amp-analytics-0.1.js"></script>
  • Attributes
    type

    https://amp.dev/documentation/guides-and-tutorials/optimize-and-measure/configure-analytics/analytics-vendors/

    googleanalytics
    refer to ga:amp
    <amp-analytics type="googleanalytics" id="analytics1">
<script type="application/json">
{
  "vars": {
    "account": "UA-XXXXX-Y"
  },
  "triggers": {
    "trackPageview": {
  "on": "visible",
  "request": "pageview"
    }
  }
}
</script>
</amp-analytics>
    attribute config (inline config)
    remote config > inline config > vendor config based on attribute type
    config

7.21.14. AMPHTML ads

<!doctype html>
<html ⚡4ads>
  <head>
    <meta charset="utf-8">
    <title>My amphtml ad</title>
    <meta name="viewport" content="width=device-width,minimum-scale=1,initial-scale=1">
    <!-- AMPHTML ads do not require a <link rel="canonical"> tag. -->

    <!-- runtime -->
    <script async src="https://cdn.ampproject.org/amp4ads-v0.js"></script>

    <!-- boilerplate -->
    <!-- rationale: hide body until AMP runtime is ready and then unhides -->

    <style amp4ads-boilerplate>body{visibility:hidden}</style>

    <!-- creative CSS is limited to 20KB -->
    <style amp-custom></style>

    <a target="_blank"
   href="https://amp.dev/documentation/examples/style-layout/banner_ad/index.html">
  <amp-img src="https://amp.dev/static/samples/img/amp-300x250.jpg"
       width="300"
       height="250"
       layout="responsive"
       alt="a4a image"></amp-img>
    </a>
  </head>
  <body>
  </body>
</html>

8. JavaScript

8.1. Loading Sequence js:defer js:async

  • Refer to html:event:DOMContentLoaded
  • <script> tags without any attributes are downloaded together and executed immediately in order
    defer
    Works when src is defined. Download together, execute in order and defer execution until before DOMContentLoaded. The download and execute process do not block HTML parsing
    async
    Works when src is defined. Download together, execute in any order as soon as they're downloaded and DOMContentLoaded does not have to wait for the execution. The download process does not stop HTML passing, but the execution process does. If async scripts are downloaded and executed before DCL, it will still block HTML parsing
    async="false"
    Works when src is defined. Download together, execute in order as soon as all are downloaded
    (no term)
    async is useful when the script files are not dependent on other files and do not have any dependencies themselves. Use it when you don't care exactly at which point the file is executed
    (no term)
    By default, all dynamically added scripts are async
    (no term)

    For example, if you want to load 2 javascript files and execute them in order but after they are all downloaded, then async="false" is needed 

      [ '1.js',
'2.js'
  ].forEach(function(src) {
var script = document.createElement('script');
script.src = src;
script.async = false;
document.head.appendChild(script);
  });

8.2. External Script, document.write js:external script

  • Loading external script use this 

    [ 'https://ajax.googleapis.com/ajax/libs/jqueryui/1.10.2/jquery-ui.min.js','/sites/all/themes/abc/js/interstitial.js'
].forEach(function(src) {
var script = document.createElement('script');
script.src = src;
script.async = false;
document.head.appendChild(script);

// another way
var s = document.scripts[0];
s.parentNode.insertBefore(script, s);
});
  • refer to css:js insert
  • document.write inside external script (asynchronously loaded) cannot modify DOM. Use these to modify the DOM. Refer to html:ElementObject
    • .appendChild()

    • .insertBefore() 

      var t = document.getElementById("childElement");

t.parentNode.insertBefore(newNode, t);

    • equivalent to jQuery .html('html')

      • Get or set the HTML or XML markup contained within the element
      Set

      remove all of the element's descendants and replace them with nodes constructed by parsing the HTML given in the string

      • <script> elements inserted using innerHTML do not execute when they are inserted
      • onclick attributes are valid
      var html = [];
// var html = <?php echo $html; ?>;
html.push(<?php echo $html; ?>);
html.push(<?php echo $html2; ?>);
// t.innerHTML = html; // html is converted to string with commas
t.innerHTML = html.join('');

    • get and set text inside the element

      Get
      result is a plain text with all HTML tags stripped
      • trigger reflow
        Set
        set text only. No HTML tag parsing
    • https://developer.mozilla.org/en-US/docs/Web/API/Node/textContent
      • concatenation of the textContent of every child node, excluding comments and processing instructions
      • Set textContent on a node removes all of the node's children and replaces them with a single text node with the given string value
    • Refer to tool:html:document.write

    • https://support.jwplayer.com/customer/portal/articles/1406723#fndtn-method-1

      • Refer to video:jw
      • <script src="//content.jwplatform.com/players/MEDIAID-PLAYERID.js"> </script>
      • If this is async loaded, the player will not show because it has document.write
      • <script src="//content.jwplatform.com/libraries/PLAYERID.js"> </script>
      And a container
      <div id="myElement"></div>
      Then dynamically load this

      TEMPLATEID is the Source file name for a video content in JW UI (under Sources). 

      <script type="text/javaScript">
  var playerInstance = jwplayer("myElement");
  playerInstance.setup({
  file: "//content.jwplatform.com/videos/MEDIAID-TEMPLATEID.mp4"
  mediaid: "xxxxYYYY"
  });
</script>

8.3. Initialization >

Always initialize variables at the top including vars in loops 

var myObj = {},
    myString = "",
    myArray = [],
    myBoolean = false,
    myRegex = /()/,
    myNumber = 0;

These values are considered as false

8.4. Variable Scope

8.4.1. Hoisting

  • See 8.6.10
  • var variables are hoisted to the top of their scope and initialized with a value of undefined
  • Only function can create local scope
    • This is called function-level scoping while PHP has block-level scoping, where new scope can be defined in block
    • Without using var or let to declare variables in any function (it's called implicitly declared) and that function runs, those variables are global
myFunction();

function myFunction() {
  carName = "Volvo";
  // carName is actually global
  var carName2 = "Freightliner";
  // carName2 is not global
  // Local var takes precedence when it's used inside a function
  // the inner var shadows the outer.
}
// Alternative way to get global var
console.log(window.carName);

// function and variable declaration are moved to the top of their
// scope (global or local). It's called hoisting
// which allows you to use them before they are declared
// Functions are hoisted first and then variables
// Function declarations have priority over variable declarations, but not variable assignments

showState("one", {}, 2, 0.98); // output: Ready

// function declaration
// Pros :: Can be used before function is declared
// Cons :: Can't conditionally declare a function
function showState() {
  console.log("Ready");
  // arguments[0] is "one"
}

// function expression/definition/assignment
// Pros :: Can conditionally define a function
// Cons :: Can't be used before it is defined 
var showState = function() {
  console.log("Idle");
};

showState(); // output: Idle

// The function that is assigned to a variable doesn't have to be anonymous

really.long.external.scoped.name = function shortcut(n) {
 // regression
 shortcut(n-1);
 // Pass it self as a callback
 someFunction(shortcut);
}

8.4.2. let, const, var

  • var
    • A variable is defined by var then this variable is available to the closest function scope
      When a var is declared outside a function
      global
      When a var is declared inside a function block
      function scoped
    • Always js:initialize for variables declared by var
    • See 8.4.1
  • let and const are block-scoped
    • let declares a variable which is available to the block it is enclosed in
    • A block is anything inside {} e.g. switch, for, if
    • Can only let declare the same variable once in the same block
      • The same variable can be let declaired inside a subblock and it will have new scope but not overwrite the upper scope variable
    • let declared variable must be declared before it is used or referenced
  • Limitation about var
    • Use let in a loop, the variable can be re-binded for each iteration

    • while var in a loop, only the final assigned value is passed unless a closure is used 

      for (var i = 1; i <= 5; i++) {
    setTimeout(
        function () {
            console.log(i);
        }, 1000 * i);
// 5 5 5 5 5

    setTimeout(
        function (x) {
            return function () {
                console.log(x);
            };
        }(i)
        , 1000 * i);
// 1 2 3 4 5
}
  • const is the same as let but it's read only. Uppercase naming
    • The same const variable name can be redeclared to different value in sub block
    • If const is an array or object, its child element or key/value is not protected by default
      • Which means you can push to const array or modify key/value
    • const in the same block cannot be changed or redeclared or redeclared by let or var

    • const can be used in js:for…of and js:for…in loops but not the imperative for loop 

      let scores = [75, 80, 95];
for (const score of scores) {
  console.log(score);
}

// throws error
for (const i = 0; i < scores.length; i++) { // TypeError
  console.log(scores[i]);
}

8.5. Closure, Javascript Singleton

8.5.1. Compare to PHP Closure php:closure

8.5.2. IIFE (Immediately Invoked Function Expression)

// function declaration
function() { console.log('test') }

// returns function(){ console.log('test') }
(function() { console.log('test'); })

function myFunc() {
    // true except when var anInstance = new myFunc();
    console.log(this === window);
    return "global function myFunc";
}

// Immediately Invoked Function Expression (IIFE) Variation 1
var abc = (function() {
    console.log(this === window);
    return 'test';
})();

console.log(abc === "test");

// Or
var abc = (function myFunc() {
    console.log(this === window);
    return 'test';
})();

console.log(abc === "test");
console.log(myFunc() === "global function myFunc");

// IIFE Variation 2
var abc = (function() {
    console.log(this === window);
    return "test";
}());

// Or
var abc = (function myFunc() {
    return "test";
}());

console.log(abc === "test");
console.log(myFunc() === "global function myFunc");

// IIFE Variation 3
// ! eq. to +, -, or even ~. Any unary operator
// returns true. because the function run returns nothing (undefined) !undefined === true
// Don't use it for assigning value to a variable
!function() {
    console.log(this === window);
}();

// return false because IIFE returns 'test' and !'test' === false
!function() {
    return 'test';
}();

// Don't do this. abc will be false
var abc= !function() { return 'IIFE'; }();

// IIFE Variation 4 (only for assigning return value to a variable)
var abc = function() {
    return 'test';
}();

// Or
var abc = function myFunc() {
    return 'test';
}();

// IIFE Variation 5. Use parent's `this` inside the IIFE
// See js:function:call

function Person() {
    this.instanceProperty = 10;
    var varVar = "a";
    let letVar = "b";

    setTimeout(function() {
        console.log('Person setTimeout callback is not an arrow function');
        console.log(this === window);
        console.log(varVar === "a");
        console.log(letVar === "b");
    }, 0);

    setTimeout(() => {
        console.log('Person setTimeout callback is an arrow function executed under the instance scope');
        console.log(this !== window);
        console.log(this.instanceProperty === 10);
        console.log(varVar === "a");
        console.log(letVar === "b");
        ;(function() {
            console.log("IIFE with 'this' overriden");
            console.log(this !== window);
            console.log(this.instanceProperty === 10);
            console.log(varVar === "a");
            console.log(letVar === "b");
        }.call(this));
        !function() {
            console.log("IIFE without overriding 'this'");
            console.log(this === window);
            console.log(varVar === "a");
            console.log(letVar === "b");
        }();
    }, 0);
}

8.5.3. Revealing Module Pattern

  • It means to use function scope, IIFE and JavaScript Object Prototype to create public and private methods
  • More on 8.5.7
  • Check another pattern 8.17
var Singleton = function () {
    var privateCounter = 0;

    function changeBy(val) {
        privateCounter += val;
    }

    // private methods
    function sayHello() {
        console.log('Hello');
    }

    // public methods
    return {
        sayHello: sayHello,
        increment: function () {
            changeBy(1);
        },
        callBothPrivatePublicMethods: function () {
            changeBy(1);
            this.increment();
        },
        getValue: function () {
            return privateCounter;
        },
        callbackExample: function () {
            [].forEach.call(
                document.getElementsByClassName('my-class'),
                function (ele) {
                    changeBy(1);
                }
            )
        },
    };
}();

// Hello
Singleton.sayHello();

Singleton.increment();
// 1
console.log(Singleton.getValue());

Singleton.callbackExample();
// 4
console.log(Singleton.getValue());

Singleton.callBothPrivatePublicMethods();
// 6
console.log(Singleton.getValue());

// Non-singleton
// Expose module as global variable
var Module = function () {

    // Inner logic
    function sayHello() {
        console.log('Hello');
    }

    // Expose API
    return {
        sayHello: sayHello
    }
};

// instantiate
var module1 = Module();
// Or
// var module1 = new Module();
module.sayHello();

8.5.4. Semi-colon

// Use this IIFE format if semi colon may or may not be enforced 
lilitest = function() { console.log('lilitest'); } // no ;
!function() { console.log('IIFE'); }() // no ;, returns true. lilitest is not run. Perfect!

lilitest = function() { console.log('lilitest'); } // no ;
!function() {}(); // with ;, returns true. lilitest is not run. Perfect!

lilitest = function() { console.log('lilitest'); }; // with ;
!function() {}(); // with ;, returns true. lilitest is not run. Perfect!

lilitest = function() { console.log('lilitest'); }; // with ;
!function() {}() // no ;, returns true. lilitest is not run. Perfect!

// For this IIFE format, always add ; to each line otherwise expect unexpected results
lilitest = function() { console.log('lilitest'); } // no ;
(function() {console.log('IIFE');})() // no ;, lilitest is run! Error. NOT GOOD

lilitest = function() { console.log('lilitest'); } // no ;
(function() {})(); // with ;, lilitest is run! NOT GOOD

lilitest = function() { console.log('lilitest'); }; // with ;
(function() {})() // no ;, lilitest is not run! Perfect!

lilitest = function() { console.log('lilitest'); }; // with ;
(function() {})(); // with ;, lilitest is not run! Perfect!

lilitest = function() { console.log('lilitest'); }; // with ;
(function() {})(); // with ;, lilitest is not run. Perfect!

// For this IIFE format, always add ; to each line otherwise expect unexpected results
lilitest = function() { console.log('lilitest'); } // no ;
(function() {console.log('IIFE');}()) // no ;, lilitest is run! NOT GOOD

lilitest = function() { console.log('lilitest'); } // no ;
(function() {console.log('IIFE');}()); // with ;, lilitest is run! NOT GOOD

lilitest = function() { console.log('lilitest'); }; // with ;
(function() {console.log('IIFE');}()) // no ;, lilitest is not run! Perfect!

lilitest = function() { console.log('lilitest'); }; // with ;
(function() {console.log('IIFE');}()); // with ;, lilitest is not run! Perfect!

// Variable assigned by the return of IIFE is fine with inconsistent semi-colons 
lilitest = function() { console.log('lilitest'); } // no ;
lilitest2 = (function() {})(); // with ;, lilitest is not run. Perfect!

lilitest = function() { console.log('lilitest'); } // no ;
lilitest2 = (function() {})() // no ;, lilitest is not run. Perfect!

lilitest = function() { console.log('lilitest'); } // no ;
lilitest2 = function() {}(); // with ;, lilitest is not run. Perfect!

lilitest = function() { console.log('lilitest'); } // no ;
lilitest2 = function() {}() // no ;, lilitest is not run. Perfect!

8.5.5. Closure does not have to be defined as a result of immediately run

function outer() {
    var b = 10;
    function inner() {
    var a = 20; 
    console.log(a+b);
    }
    return inner;
}
var X = outer();

8.5.6. Syntax, When to use closure?

  • A closure gives access to an outer function's scope from an inner function
    • Closures are created every time a function is created, at function creation time
    • Closure is a function bundled together (enclosed) with references to its surrounding state (lexical environment)
  • When you don't want to change the variables in outer scope where the closure is in
    • The outer scope can be global or any place where the closure is in
  • When you want to run the function later
  • When you want to run the function later by providing just reference to other function
  • When you want to run the function later by providing initial state
  • When you want to do regression, don't want to affect outer scope and need more debugging
(function() {
  // Codes here runs immediately
})();

(function($) {
  // Use jQuery with the shortcut
  console.log($.browser);
  $(document).ready(function($) {
    // document ready
   }
  );
}(jQuery));

// Another form
jQuery(document).ready(function($) {
  // Use $ now
});

8.5.7. Singleton 

var Managers = {}; //namespace
// Singleton object CssManager
// `Managers` and singleton object `CssManager` has to be defined before they are used

Managers.CssManager = (function () {

    //region Private Space

    // All variables and methods are protected from the global scope

    // To initialize, assign aliases of global properties/methods/functions/objects for this Singleton
    var doc = document;

    // Private members can have the same name as in the object and they are referred diffrently inside the object
    // - privateAndMember();
    // - this.publicMember();

    var setAttributes = function (element, attributes) {
        for (attribute in attributes) {
            element[attribute] = attributes[attribute];
        }
    }
    //endregion

    return {
        addStyleSheet: function (id, url) {
            // global variable `document` can be used, or use alias `doc` as defined above in Private Space
            var newStyleSheet = document.createElement("link");
            // call private member
            setAttributes(newStyleSheet, {
                rel: "stylesheet",
                type: "text/css",
                id: id,
                href: url
            });
            document.getElementByTagName("head")[0].appendChild(newStyleSheet);
        },
        removeStyleSheet: function (id) {
            var currentStyleSheet = document.getElementById(id);
            if (currentStyleSheet) {
                currentStyleSheet.parentNode.removeChild(currentStyleSheet);
            }
        },
        swapStyleSheet: function (id, url) {
            // call other public member
            this.removeStyleSheet(id);
            this.addStyleSheet(id, url);
        }
    }

})();

8.5.8. Regression

This provides you more debugging info 

var charsInBody = (function counter(elm) {
  // Notice :: this is not an anonymous function as we see before in a closure!
  if (elm.nodeType == 3) {
    return elm.nodeValue.length;
  }
  var count = 0;
  for (var i = 0, child; child = elm.childNodes[i]; i++) {
    count += counter(child);
  }
  return count;
})(document.body);

8.6. Types, typeof, constructor

8.6.1. Number

Used in
js:Math
IsNaN
true if it is not a number
var foo = "55";
var myNumber = Number(foo);
Number(' 55  ') === Number('55.0') === 55
Number.isInteger(Number(' 55.00 ')) === true

if (!isNaN(myNumber)) {
  console.log("It is a number");
  // foo is a number
}

console.log( Number.parseFloat(x).toFixed(2) ); // round up sometimes not working using toFixed..

let decimals = 2;
console.log( Number(Math.round(value +'e'+ decimals) +'e-'+ decimals).toFixed(decimals) ); // use this to round

function isInt(value) {
  return !isNaN(value) && (function(x) { return (x | 0) === x; })(parseFloat(value))
}

isInt(42)        // true
isInt("42")      // true
isInt(4e2)       // true
isInt("4e2")     // true
isInt(" 1 ")     // true
isInt("")        // false
isInt("  ")      // false
isInt(42.1)      // false
isInt("1a")      // false
isInt("4e2a")    // false
isInt(null)      // false
isInt(undefined) // false
isInt(NaN)       // false

8.6.2. Boolean

  • true or false. NOT TRUE not FALSE
  • "" 0 false null undefined

8.6.3. undefined undefined or has not been assigned a value

// Variable
let name;

if (name === undefined) {
    console.log("This is undefined");
}

// Object
let user = {
    name: "John Doe",
    age: 14
};

if (user.hobby === undefined) {
    console.log("This is undefined");
}

// Array
let scores = [12, 34, 66, 78];

if (scores[10] === undefined) {
    console.log("This is undefined");
}

if(typeof user.hobby === "undefined"){}
if(typeof scores[10] === "undefined"){}
if(typeof name === "undefined"){}

8.6.4. null - a special type of object

8.6.5. String

  • See 8.6.1 to convert a string to number
s.length;
s.charAt(s.length-1);
s.charCodeAt(s.length-1); // UTF-16 code unit value
var myString = String.fromCharCode(65,66,67); // 'ABC'
s.indexOf("Paul"); // Case sensitive. Starts from the beginning. -1 not found
s.lastIndexOf("Paul"); // Case sensitive. Starts from the end
s.substr(0,4); // From start to 4th index
s.substring(0,4); // From start and take 4 characters. If stop is omitted, take the rest

s.slice(0,4); // Same as substring but start and stop can be both or either negative
// If start is negative, start from the end of string
// If stop is negative, stop to (s.length - 1) - Math.abs(s)

s.toLowerCase();
s.toUpperCase();

// Case insenstive comparison
// Don't use string1 == string2
// Use below for comparing UTF8
if ( string1.toUpperCase() === string2.toUpperCase() ) {}

// null converted to string is 'null'
var abc = null;
var abcDisplay = 'abc: ' + abc;
console.log(abcDisplay); // "abc: null"
8.6.5.1. Padding 0's to left
function padLeft(nr, n, str){
  return Array(n-String(nr).length+1).join(str||'0')+nr;
}
var mystring = 3;
console.log(padLeft(mystring,3)); // 003
console.log(padLeft(mystring,3,'z')); // zz3
8.6.5.2. Template Literal
  • https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals

  • Basics 

    // Template literal. Since js:es6
console.log(`This is ${myVar} times easier!`);
let a = 5;
let b = 10;
console.log(`Fifteen is ${a + b} and
not ${2 * a + b}.`);
// need to escape backtick
console.log(`Escape \``);
// multi lines
console.log(`string text line 1
string text line 2`);
// "string text line 1
// string text line 2"

  • Inside an HTML's <script> tag, and there's </script> inside it, no matter if it's a string in quotes, apostrophes, or even a template literal, it will always close the script tag. Escape it: 

    <script>
  var str = `
  <div>

  </div>
  <script>
  <\/script>
`
    var pre = document.createElement('pre')
    pre.textContent = str
    document.body.appendChild(pre)
</script>

8.6.6. Array

8.6.6.1. Array.xxx() vs Array.prototype.xxx()

Array.xxx() is static method while Array.prototype.xxx() is instance method

8.6.6.2. DOM elements HTMLCollection
  • HTMLCollection is not an array. Need to use
    • Array.prototype.reduce.call(eles, function(a, b) {}, initialValue)
    • [].reduce.call(eles, function (a,b) {}, initialValue)
8.6.6.3. Numeric array key exists
var a = [1,2];
if (a[8] !== undefined) {
    console.log(a[8]);
}
8.6.6.4. Variable is a non-empty array
var a = {
    xyz: "abc"
};
var myVar = a.abc;

console.log((typeof notDeclared === "undefined") === true);
console.log((Array.isArray(myVar) && myVar.length) === false);

// Use this shortcut if type does not matter
console.log(myVar?.length === undefined);
console.log(a.abc?.length === undefined);
console.log(a.xyz?.length === 3);
8.6.6.5. array.indexOf(), array.prototype.includes()

  • array.indexOf() 

    arr.indexOf(searchElement[, fromIndex])

var a = [1,2,3];
a.indexOf(4); // -1
a.indexOf(1); // 0

(['joe', 'jane', 'mary'].indexOf('jane') >= 0; // true
    • All browsers support
    • ===

  • array.prototype.includes() 

    arr.includes(valueToFind[, fromIndex])

['joe', 'jane', 'mary'].includes('jane'); // true
    • IE not supported
8.6.6.6. Array.prototype.pop(), Array.prototype.shift(), array.push(), array.unshift()
a.pop(); // remove last element
a.shift(); // remove fist element, mutable
a.push("Z"); // add to the end of array. return new length
a[a.length]="Z"; // also append to last

a.unshift("Z"); // add to the beginning of array. return new length
8.6.6.7. array.concat() merge without altering nor removing dups, return a new array
var a = new Array("P", "C", "S");
var b = new Array(31,29,34);
a = a.concat(b); // Join arrays P, C, S, 31, 29, 34

// remove dup
function removeDuplicate(arr){
  var exists ={},
  outArr = [], 
  elm;

  for(var i =0; i<arr.length; i++){
    elm = arr[i];
    if(!exists[elm]){
  outArr.push(elm);
  exists[elm] = true;
   }
  }
  return outArr;
}

// e.g. removeDuplicate([1,3,3,3,1,5,6,7,8,1]);
// = [1, 3, 5, 6, 7, 8]

// if you want mutation instead of creating a new array
var a = ['a','b'];
var b = ['a', 'x', 'y'];
a.push.apply(a,b); // cross-browser
//Array.prototype.push.apply(a,b); // cross-browser
// a.push(a, ...b); // NodeJS where ES6/ES2015 is supported
console.log(a);
// ["a", "b", "a", "x", "y"]
8.6.6.8. array.slice() vs array.splice()
  • Use js:array:filter to remove values from array instead if mutation is not desired
Array.prototype.slice([start[, end]])
// not mutated
// start :: from 0
// end :: from 0. Up to this index but not including

var a = [1,2,3];
var slice = a.slice(1,2); // [2]

let arrDeletedItems = array.splice(start[, deleteCount[, item1[, item2[, ...]]]])
// mutated!
// start :: from 0
// deleteCount :: An integer indicating the number of elements in the array to remove from start. If deleteCount is omitted, or if its value is equal to or larger than array.length - start (that is, if it is equal to or greater than the number of elements left in the array, starting at start), then all the elements from start to the end of the array will be deleted.
// item1, item2, ... Optional :: The elements to add to the array, beginning from start. If you do not specify any elements, splice() will only remove elements from the array.

var deletedItems = a.splice(1,2); // deletedItems = [2,3] and a becomes [1]

// remove values in array
var arr = [1, 2, 3, 4, 5, 5, 6, 7, 8, 5, 9, 0];
for( var i = 0; i < arr.length; i++){
    if ( arr[i] === 5) {
    arr.splice(i, 1);
    i--;
    }
}
//=> [1, 2, 3, 4, 6, 7, 8, 9, 0]
8.6.6.9. array.join(), array.toString(), array.valueOf()
var join = a.join(","); // "P,C,S"
a.toString() // also "P,C,S"
a.valueOf(); // also "P,C,S". Convert to primitive
8.6.6.10. array.sort(), array.reverse()
var sort = a.sort(); // sort as strings. case sensitive: C, P, S. Mutable
function numericSort(anArray) {
 anArray.sort(function(a,b) {
  return a - b; // ascending order
  // return a.year - b.year; if it's an obj
});
 return anArray;
}

function randomSort(anArray) {
 anArray.sort(function(a, b) {return 0.5 - Math.random();});
 return anArray;
}

var reverse = a.reverse(); // S, C, P
8.6.6.11. array loop: for, array.forEach()
for (var i = 0; i < fruits.length; i++) {
 fruits[i];
}

js:forEach 

fruits.forEach(function(i) {
 console.log(i);
});
8.6.6.12. array.map(), array.filter()
  • Do something while each item is iterated, join returned values into an array
  • Use js:array:splice to remove values from array if mutation is desired
var result = a.map(function(i) { return i.toUpperCase(); }); // result is a new array
console.log(a); // a is not affected!

// array.filter
// not mutated
a = a.filter(function(i) { 
   if ( true then remove condition ) { 
    return false; 
   }
   else {
    return true; // keep
   }
});


var array = [1, 2, 3, 4, 5, 6, 7, 8, 9, 0];
var filtered = array.filter(function(value, index, arr){
    return value > 5;
});
//filtered => [6, 7, 8, 9]
//array => [1, 2, 3, 4, 5, 6, 7, 8, 9, 0]
8.6.6.13. array.every(), array.some()
  • See 2
// arr.every(callback(element[, index[, array]])[, thisArg])

a.every(function(i) { return i[0] === "a"; });
// return true only if all returns are true
// If array is empty, it returns true. Does not matter what condition


a.some(function(i)  { return i[0] === "a"; });
// return true if at least one return is true
8.6.6.14. Array.prototype.reduce()
var sum = [0, 1, 2, 3].reduce(
 function(a,b) { return a + b; },
 0
);
var sum2 = [...Array(4).keys()].reduce(
    // eq. for (let i = 0; i <= 3; i++)
    // eq. [0, 1, 2, 3]
    function(a, b) {
        return a + b;
    },
    0
);
// sum === sum2 === integer 6

var flattened = [[0, 1], [2, 3], [4, 5]].reduce(
 function(a,b) { return a.concat(b); },
 []
);
// [0, 1, 2, 3, 4, 5]

array.reduce(callback[, initialValue])

  • If initialValue is provided, it will start at index 0
    • If not provided, it will start at index 1 and previousValue is the value at index 0 in the array
  • callback has 4 arguments
    • previousValue
    • currentValue
    • currentIndex
    • the input array was called upon

8.6.7. Date 

var date1 = new Date();
var date2 = new Date(77978); // milliseconds since 1970 01 01 GMT
var unixTimestampFromPHP = parseInt( (date1.getTime()/1000).toFixed(0) );
date2 = new Date( unixTimestampFromPHP * 1000 );

var date3 = new Date(2017, 0, 31, 15, 35, 20, 20); // 2017 Jan 1 at 15:35:20 at 20 ms

// ISO-8601 format works across browser
var date5 = new Date('2011-04-11T10:20:30Z');

// Don't use any format other than the above. This does not guarantee to work
var date4 = new Date("31 January 2016"); 

// Get and Set
var dayofmonth = date4.getDate(); // setDate
var dayofweek = date4.getDay(); // int, Sunday as 0. You can't set day
var month = date4.getMonth(); // int, January as 0. setMonth
var fullyear = date4.getFullYear(); // 4 digits. setFullYear
var datestring = date4.toDateString(); // Current time zone. "Wed 31 Dec 2003"

var currentDay = date4.getDate();
date4.setDate(currentDay + 28 ); // Plus 28 days

// getHours(), getMinutes(), getSeconds(), getMilliseconds(), toTimeString()
// setHours(), setMinutes(), setSeconds(), setMilliseconds()

// Date difference
var date1 = new Date();
var date2 = new Date('01-31-2016');
var timeDiff = date2.getTime() - date1.getTime(); // milliseconds in UTC
var diff Days = timeDiff / (1000*3600*24);

date1.getUTCFullYear() + '-' + 
String(date1.getUTCMonth() + 1).padStart(2, '0') + '-' + 
String(date1.getUTCDate()).padStart(2, '0') + '-' + 
    String(date1.getUTCHours()).padStart(2, '0') + '-' +
    String(date1.getUTCMinutes()).padStart(2, '0') + '-' +
    String(date1.getUTCSeconds()).padStart(2, '0'); // YYYY-MM-DD-HH-mm-ss in UTC
8.6.7.1. Date to string
date4.toISOString(); // e.g. 2011-10-05T14:48:00.000Z
// <<js:time:iso>>
// timezone is always zero UTC offset. 24 or 27 (I haven't seen one..) characters.
// Good for php:f:strtotime

date4.toUTCString(); // UTC timezone. Sun, 23 Aug 2020 03:03:10 GMT
date4.toGMTString(); // same as .toUTCString()
date4.toString(); // browser's timezone. Sat Aug 22 2020 23:03:10 GMT-0400 (Eastern Daylight Time)

date4.toLocaleString(); // use browser's default locale
date4.toLocaleDateString(); // use browser's default locale

// My browser's default locale is en-US

// Use a specific locale
date4.toLocaleString( 'en-US' ); // 8/22/2020, 10:01:19 PM
date4.toLocaleDateString( 'en-US' ); // 8/22/2020

// Use options
var options = {
    timezone: 'America/Toronto'
};

date4.toLocalDateString( undefined, options ); // Use default locale, and use custom options

options = {
    timezone: 'America/Toronto',
    month: 'short',
    day: 'numeric',
    year: 'numeric'};
date4.toLocaleDateString( 'en-US', options ); // Aug 22, 2020
8.6.7.2. Compare current date with a future date
function checkFuture(futureDateString) {
  var futureMS = new Date(futureDateString).getTime();
  var currentMS = new Date().getTime();

  if (currentMS >= futureMS) {
    return true; // current date is beyond future date
  }
  return false;
}
checkFuture("01/03/2018"); // mm/dd/yyyy

8.6.8. Math js:Math

Math.PI
Math.abs(-101);
Math.ceil(101.01); // 102
Math.floor(101.01); // 101
Math.round(101.01);
Math.pow(10, 2); // 100
parseInt("10"); // 10

var itemCost = 9.99 * 1.075; // 10.73925
itemCost.toFixed(2); // 10.74

8.6.9. RegExp

8.6.9.1. search(), replace(), match()
var s = "Visit W3Schools!";
var n = s.search("W3Schools"); // 6. -1 if not found
var all = s.match("i"); // return all matches
var res = s.replace(/w3schools/i, "Microsoft");
var res = s.replace("W3Schools", "Microsoft");
8.6.9.2. RegExp
  1. Ways to define regex 
    var n = s.search(/w3schools/i); // Use forward slash. Return the first matched index
var pattern = /w3schools/i; // Define a pattern in a variable

// Need to escape \ with \\
var patternobj = new RegExp('w3schools','i'); // Define a pattern as a RegExp object 
var patternobj = new RegExp(/w3schools/,'i');

// Test a pattern
pattern.test("Perform a test on this string"); // true or false

// Return the first found text
pattern.exec("Perform an exec on this string"); // null if not found.

// Index
every time pattern.exec() or pattern.test() is run, the pattern.lastIndex is advanced.
Until .lastIndex gets to 0

// Return all matches
s.match(pattern);
  2. Pattern and modifiers
    Syntax
    /pattern/modifiers
    (no term)
    Modifiers
    • PHP PCRE Pattern Modifiers
    • case-insensitive
    • find all match
    • multiline
    • ungreedy
    • the dot metacharacter matches all characters including newlines
    (x|y)
    one of x or y
    (no term)

    Quantifier

    n+ at least one of n
    n* zero or more
    n? zero or one
    n{X} \d{4} a 4-digit string
    n{X,Y} \d{3,4} a 3-to-4-digit string
    n{X,} \d{3,} a at-least-3-digit string
    n$ at the end
    ^n at the beginning
    (?=n) non greedy match
  3. Character Class or Character Set
    Brackets
    everything inside a pair of brackets are character class
    • https://www.regular-expressions.info/refcharclass.html
    • Refer to linux:regex:character class for special character classes that can only be used inside []
      • Some languages don't allow js:regex:metacharacter inside [], so special character classes are the only choice
    • Examples
      [abc]
      a character that is a, b or c
      [^abc]
      a character that is not a, b nor c
      [0-9]
      a digit
      [^0-9]
      a non-digit character
      Alphanumeric
      [0-9a-zA-Z]
    • [base-[subtract]]
      [a-z-[aeiuo]]
      matches a single letter that is not a vowel
    • [base&&[intersect]]
      [a-z&&[^aeiuo]]
      matches a single letter that is not a vowel
    • use backslash \ to escape
      • non-literal characters
        • ^
        • as it's used in range and subtraction
        • ]
        • /
      • Characters in special occasions
        \
        escape backslash itself
        *
        no need to escape inside []
        .
        no need to escape inside []. Except in Nginx
  4. Metacharacter
    • Some language can use metacharacters inside square brackets, some don't allow
      • For those that don't allow, use special character classes inside []. Refer to [[linux:regex:character class][linux:regex:character class
    . a character execpt newline or line terminator
    \w a word character including _ [A-Za-z0-9_]
    \W a non-word character. same as [$\w]
    \d a digit
    \D a non-digit character
    \s whitespace
    § a non-space character
    \b word boundary
    \0 a NUL character
    \n a new line character
    \f a form feed character. e.g. page/section break
    \r a carriage return
    \t a tab character
    \v a vertical tab character
    \uxxxx a Unicode character \u0057 is w
    \xxx \127 is w
    \xdd \x57 is w
       
    1. Word Boundary \b and \B
      var s = "Visit W3Schools";
var pattern = /\bW3/g; // return words that start with W3
pattern.exec(s); // W3
var patter = /\bVisit\b/g; // whole word
var patterB = /\BSchools/g; // return "Schools". Words that do not start or end with Schools
  5. Non greedy match (?=n)
    var s = 'Is this all there is';
var p = /is(?= all)/g;
s.match(p); // is
// (?= all) matches any string that is before ' all':  but the matched is not returned.
// The other way to intepret it is find any 'is' that has ' all' right behind
var p = /is(?! all)/g; // Find any 'is' that doesn't have ' all' right behind
  6. Match HTML Tags regex:match_tags

    Match all tags 

    '<[^>]*>'

    Match one tag only 

    '<img[^>]*>'
'<(img|/?div|br)[^>]*>'

8.6.10. Function

8.6.10.1. arguments
function add() {
  var sum = 0;
  for (var i = 0, j = arguments.length; i < j; i++) {
    sum += arguments[i];
  }
  return sum;
}
add(2,3,4,5);

// Use Spread Syntax js:spread
function add(firstValue, ...args) {
  var sum = 0;
  for (let value of args) {
    sum += value;
  }
  sum += firstValue;
  return sum;
}
add(2,3,4,5);

// Optional argument
function verify(w) {
  w = (typeof w === 'undefined') ? 'default' : w;
}
8.6.10.2. Default parameter
// Not supported: IE, Safari < 10
function multiply(a, b = 1) {
    return a * b;
}

// backward compatible
function abc(a){
    if(a === undefined){
    a = 2;
    }
}
8.6.10.3. .apply(), .call()
.apply()
Pass arguments as an array to a function which accepts multiple arguments
  • funct.apply(thisArg, [argsArray])
  • pass thisArg into funct as this in funct
  • array or array-like object
  • Pass no more than 65536 arguments/elements in array
.call(thisArg, arg1, ... , argN)
Pass arguments as arguments to a function which accepts multiple arguments
thisArg
pass it into funct as this in funct
(no term)
See IIFE
var numbers = [5,6,2,3,7];
var max = Math.max.apply(null, numbers); // Equivalent to Math.max(5,6, ...);
var max = Math.max.call(null, 5,6,2,3,7);

They are also used to call a method on a varible (obj or array) which doesn't have that method as its property.

  • Loop arrays which don't have .forEach() such as NodeList
8.6.10.4. .bind() 
var cb = function(rq) {
    this.successCB(rq);
    this.failCB(rq);
};
cb = cb.bind({successCB: cbSuccess, failCB: cbFail});
aNormalFunctionRunsACallback(cb);

8.6.11. Exception, Error 

try {
    throw "Too big"; // Manually throw an error
    var err = new Error("Too big");
    throw err;
}
catch (err) {
    // err === 'Too big' if it's a string
    console.log(
    err.message, // Standard
    err.description, // Microsoft.
    err.number, // Microsoft. same as linenumber
    err.linenumber, // FF and Chrome
    err.name, // Standard. Initial is Error but you can change it
    err.stack // Not standard but works in all browsers.. Stack trace.
    );
}
finally {
    // success or failure, run this
}

8.6.12. Prototype Object js:prototype

  • Every js:Object has a prototype object
  • prototype object's default properties and methods
    constructor
    property. get/set
    Object.getPrototypeOf(anObj)
    e.g. Object.getPrototypeOf(document.getElementById('abc'))
    (no term)

    Get all keys (not only enumerable) in an object including prototype-chain properties and non-enumerable properties 

    function logAllProperties(obj) {
 if (obj == null) return; // recursive approach
 console.log(Object.getOwnPropertyNames(obj));
 logAllProperties(Object.getPrototypeOf(obj));
}
logAllProperties(my_object);
    • Compared to js:Object.keys which only gets own enumerable properties
    hasOwnProperty('aProperty')

    bool

    objA.hasOwnProperty('aProperty')
    not objA.prototype.hasOwnProperty('aProperty')!
    • aProperty has to be direct (not objA.prototype.aProperty) and not inherited
    • Use if ('key' in myObj) to check all enumerable keys js:for…in
    isPrototypeOf()
    js:prototype.isPrototypeOf
    • Parent.prototype.isPrototypeOf(SubParent.prototype) is true
    • Whether or not SubParent has some prototype properties removed, added or modified
    • js:instanceof
    propertyIsEnumerable
  • js:inheritance
  • Also, if any prototype property is modified in the upper chain, the lower chain and its instantiated objects will be modified, too!

8.6.13. Symbol js:symbol

  • Since js:es6
  • A symbol value is created by invoking function Symbol, which dynamically produces an anonymous, unique value
// here are two symbols with the same description,
let Sym1 = Symbol("Sym");
let Sym2 = Symbol("Sym");

console.log(Sym1 == Sym2); // returns "false"
// Symbols are guaranteed to be unique.
// Even if we create many symbols with the same description,
// they are different values.

alert(Sym1); // TypeError: Cannot convert a Symbol value to a string

alert(Sym1.toString()); // now it works

alert(Sym1.description); // Sym

const id = Symbol();
const courseInfo = {
  title: "ES6",
  topics: ["babel", "syntax", "functions", "classes"],
  id: "js-course"
};

courseInfo[id] = 41284;
console.log(courseInfo); // courseInfo has a unique key `Symbol()` which is non-representable, the value of that key is 41284

  • Symbol can be used to extract an object's iterator function 

    var title = 'ES6';
console.log(typeof title[Symbol.iterator]);

var iterateIt = title[Symbol.iterator]();
console.log(iterateIt.next());
console.log(iterateIt.next());
console.log(iterateIt.next());
console.log(iterateIt.next());

function tableReady(arr) {
  var nextIndex = 0;
  return {
next() {
  if(nextIndex < arr.length) {
    return {value: arr.shift(), done: false}
  } else {
    return {done: true}
  }
}
  }
}

var waitingList = ['Sarah', 'Heather', 'Anna', 'Meagan'];
var iterateList = tableReady(waitingList);

console.log(`${iterateList.next().value}, your table is ready`);
console.log(`${iterateList.next().value}, your table is ready`);
console.log(`${iterateList.next().value}, your table is ready`);
console.log(`${iterateList.next().value}, your table is ready`);
console.log(`Is this finished? ${iterateList.next().done}`);

8.6.14. Type Conversion

Javascript conversion is automatic. var y = "5"; var x = + y;

General methods: String(x) or x.toString() / convert to string Number(x) / convert to numbers

8.6.14.1. Number to String

x.toString(); x.toExponential(2); / 2 decimal points: 9.66e+0 x.toFixed(2); / 2 decimal points x.toPrecision(); // total number of digits

8.6.14.2. String to Number

parseInt("10"); / 10 parseInt("10.33"); / 10 parseInt("10.53"); / 10 parseInt("10 20 30"); / 10 parseInt("10 years"); / 10 parseInt("year 10"); / NaN

The same for parseFloat.

Always provide a base number :: parseInt("10", 10); // decimal base

8.7. Ternary 

// Ternary
var myVar = document.getElementById('myCheckbox');
myVar ? myVar.checked : null

// But Elvis is not supported
myVar ?: null

// May consider chaining to see if deeper key is defined
// undefined if either a or b are null/undefined, otherwise a.b.c (a has to be defined, otherwise error)
const myVar = a && a.b && a.b.c;

// eq. to the new Optional Chaining Operator `?.` (IE not supported) (a has to be defined, otherwise error)
const myVar = a?.b?.c;

// Nullish Coalescing operator `??` can also be used to set default value (IE not supported) (a has to be defined, otherwise error)
const myVariable = a?.b?.c ?? 'Some other value';

// Evaluates to 'Some other value'
const myVariable2 = null ?? 'Some other value';

// Evaluates to ''
const myVariable3 = '' ?? 'Some other value';

// Exmaple about DOM
var myCheckbox = document.getElementById('myCheckbox')?.checked;

var postData = [];
if (myCheckbox !== undefined) {
    postData.push('myCheckbox=' + (myCheckbox ? 1 : 0));
}

8.8. Loops and Breaks

8.8.1. for, while, do…while

var cars = ['a', 'b', 'c'];
for (var i = 0, len = cars.length, text = ""; i< len; i++) {
 // Loop through all properties of Array.prototype
 // 3rd party libraries or frameworks might add methods or properties to Array.prototype
 // So this is not recommended for looping an Array
}
for (var i = 0; i < cars.length; i++) {
}

while () {
}

do {
}
while (condition);

// Loop child elements, all children elements
var investTypesDiv = document.getElementById('MA_lead_final_investment_type');
var investAllTypes = [];
for (let i = 0; i < investTypesDiv.children.length; i++) {
    investAllTypes.push(investTypesDiv.children[i]);
}

8.8.2. for…in js:for…in

  • Loop through all enumerable properties of an object including prototype-chain properties
  • Can be used to loop through arrays
  • The retrieval order is not the same as the object define order! For both objects and arrays
var myObj = {};
for (var prop in myObj) {
 console.log(myObj[prop]);
}

let myArray = ["a","b","c"];
for (let key in myArray) {
    // key is a string!
    let value = myArray[key];
    console.log("key: %o, value: %o", key, value);
}

8.8.3. for…of js:for…of

  • js:for…in interates over all enumerable properties of an object including prototype-chain properties
  • js:for…of iterates over elements of any collection that has a [Symbo.iterator] property
    interates over iterable objects

    Array, Map, Set, String, TypedArray, arguments but not objects!

    • Use it with Object.keys(myObj) to loop through object keys
    const keys = Object.keys(fruits)
for (const key of keys) {
    console.log(key)
}
    • Object.keys(myObj) returns an array of enumerable own properties not including prototype-chain properties. Supported by all browsers
    • Object.values(myObj) returns an array of values
    • Object.entries(myObj) returns [ ["name","John"], ["age",30] ]
      • IE not supported

      • Example 

        const usersMap = {
    id_1: {
        name: 'John 1',
    },
    id_2: {
        name: 'John 2',
    },
};
const allUsersWithId = Object.entries(usersMap);

allUsersWithId.forEach(([userId, user]) => {
  console.log(userId, user);
});
    • Refer to js:prototype 
    const elements = document.querySelectorAll('.foo');

for (const element of elements) {
    element.addEventListener('click', doSomething);
}
    (no term)
    Order is the same as the definition!
    (no term)
    Since ES2015
for (let value of iterable) {
    console.log(value);
}

// loop Map or Set
var topics = new Map();
topics.set('HTML', '/class/html');
topics.set('CSS', '/class/css');
topics.set('JavaScript', '/class/javascript');
topics.set('Node', '/class/node');

for (let topic of topics) {
    console.log(topic); // ['HTML', '/class/html']
}

for (let topic of topics.keys()) {
    console.log(topic, "is the course name"); // 'HTML'
}

for (let topic of topics.values()) {
    console.log("The course description can be found at", topic); // '/class/html'
}

for (let course of topics.entries()) {
    console.log(course); // ['HTML', '/class/html']
}

8.8.4. Array.prototype.forEach(callback[, thisArg]) js:forEach

  • Arguments
    thisArg
    Pass a variable outside of .forEach() and change it inside .forEach() using this
    callback
    3 arguments
    • currentValue
    • index
    • array
    Return
    undefined. So .forEach() is not chainable

  • For objects that have no reference to Array.prototype e.g. document.getElementsByClassName('myClass') 

    [].forEach.call(elements, function(ele) {
    // Do something
});
  • Can't break compared to traditional loop
  • Elements appended after the call to forEach() begins will not be visited
  • The value of element is changed before forEach() visits, it will reflect the change
  • Elements removed before being visited will not be visited

8.8.5. Array.prototype.every()

  • Test whether all elements in the array pass the test implemented by the provided function. It returns a Boolean value.
  • Can break but not using break
// Inline callback function
every(function (element) { /* … */ })
every(function (element, index) { /* … */ })
every(function (element, index, array) { /* … */ })
every(function (element, index, array) { /* … */ }, thisArg)

let arr = [1, 2, 3, 4];
arr.every((elem, index, arr) => {
    arr[index + 1]--;
    console.log(`[${arr}][${index}] -> ${elem}`);
    return elem < 2; // if it is false, every() will return false and stop
});

// 1st iteration: [1,1,3,4][0] -> 1
// 2nd iteration: [1,1,2,4][1] -> 1
// 3rd iteration: [1,1,2,3][2] -> 2

8.8.6. Array.prototype.some()

  • Similar to every(), but only one item/element must pass the test.

8.8.7. break and continue

  • break
    • without a label reference, can only jump out of a loop or a switch
    • With a label reference, break can jump out of any code block
  • with or without a label, can only skip one loop iteration

A code block is a block of code between { and } 

list: {
 text += cars[0] +  "<br>";
 text += cars[1] +  "<br>";
 break list;
 text += cars[3] +  "<br>";
}

8.9. Enumerable js:Enumerable

Except inherited properties, all methods and properties are enumerable in js:for…in

8.10. JSONP, JSON

8.10.1. JSONP 

<script type="text/javascript" src="http://b.com/returnjsondata?callback=mycallback"></script>

Where b.com/returnjsondata returns a json object. 
And the whole <script> in yourwebsite.com will become 
mycallback({foo:'bar'});

8.10.2. JSON

  • JSON object names require double quotes
  • String are wrapped in double quotes
  • Boolean is true or false in lowercase
  • null can be used
  • MIME type: application/json
  • var obj = JSON.parse(jsonString);
  • obj._embedded['wp:featuredmedia'][0]
8.10.2.1. Deep Keys
// see if test.level1.level2.level3 key exists
var r1 = (((test || {}).level1 || {}).level2 || {}).level3;
if (typeof r1 !== 'undefined') {
}

8.11. DOM Manipulation

8.11.1. If an element exists

var e = document.getElementById("abc");
if (!!e) {
 // exists
}

var e = document.getElementsByClassName('aClass');
if (e.length > 0) {
  // exists
}

8.11.2. Loop querySelectorAll

var divs = document.querySelectorAll('.aClass');
[].forEach.call(divs, function(div) {
  div.style.color="red";
});
// Or
divs.forEach(function() {
    //
});

8.11.3. Remove a class

  • classList returns DOMTokenList object
  • Don't need to check existance
ELEMENT.classList.remove("CLASS_NAME");
ELEMENT.classList.add('hint');
e.classList.toggle('CLASS_NAME');
e.classList.contains('CLASS_NAME'); // bool

8.11.4. Selected Option

var e = document.getElementById("selectElementID");
var i = e.selectedIndex;
var ep = e.options[i];

ep.value;
e.value; // same as ep.value

ep.text;

8.11.5. Meta tag

this.ncmGoogleTagManager = this.ncmGoogleTagManager || {};

var ns = this.ncmGoogleTagManager;

ns.getMetaTagByProperty = function (property) {
    var metas = document.getElementsByTagName('meta');
    var content = [];

    for (i = 0; i < metas.length; i++) {
    if (metas[i].getAttribute('property') == property) {
        content.push(metas[i].getAttribute("content"));
    }
    }
    return content;
};

ns.ArticlePubTime = ns.getMetaTagByProperty('article:published_time');
if (ns.ArticlePubTime.length) {
    return ns.ArticlePubTime[0];
}
else {
    return null;
}

8.11.6. Element data attributes

<div id="id" data-primary="abc"></div>

<link rel='shortlink' href='https://www.a.ca/?p=123' />

var e = document.getElementById('id');
if (e.dataset.hasOwnProperty('primary')) {
    print e.dataset.primary;
    e.dataset.primary = 'new primary';
}

if ( i = document.querySelector("link[rel=shortlink]")) {
    console.log(i.getAttribute('href'));
}

8.11.7. Remove all children

var e = document.getElementById('abc');
while (e.firstChild) {
    e.removeChild(e.firstChild);
}

var node = document.createElement('img');
node.src = 'https://a.ca/b.png';
e.appendChild(node);

8.12. URI and URI Component Encode, Current URL, open, email link

8.12.1. encodeURIComponent vs encodeURI 

var url1 = "http://abc.com";
var url2 = "http://xyz.com/test.php?url=" + encodeURIComponent(url1);

var array1 = [1, 2, 3];
encodeURIComponent(JSON.stringify(array1));
// decodeURIComponent();

// When you want to encode a whole URL that you don't trust
var uri = "http://abc.com/folder name with space/file name with space.asp?abc=has space"
var uri_en = encodeURI(uri);
// decodeURI(str)

// encodeURI DO NOT encode these but encodeURIComponent does: @+/=&#$,:;?

// Neither encodeURI nor encodeURIComponent encodes: '*_-.()~!

// Both encodeURI and encodeURIComponent encode: %"{}<>Space

8.12.2. Current URL 

console.log(window.location.href); // whole URL

// in an iFrame, get the parent window url: window.parent.location

// window.location.href === window.location.toString() === 'http://abc.com/xyz/ijk.php?m=123#def'

// window.location.href = window.location.protocol + "//" 
// + window.location.hostname (abc.com)
// + window.location.pathname (/xyz/ijk.php, if homepage, '/')
// + window.location.search (?m=123)
// +  window.location.hash; (#def)

// click to email
function sendMail() {
    var link = "mailto:me@example.com"
    + "?cc=myCCaddress@example.com"    
    + "&subject=" + encodeURIComponent("This is my subject")
    + "&body=" + encodeURIComponent(document.getElementById('myText').value);

    window.location.href = link; // redirect
}

// open a new tab
window.open('/your-url', '_blank');

// Get URL Parameter
var getUrlParameter = function getUrlParameter(sParam) {
    var sPageURL = decodeURIComponent(window.location.search.substring(1)),
    sURLVariables = sPageURL.split('&'),
    sParameterName,
    i;

    for (i = 0; i < sURLVariables.length; i++) {
    sParameterName = sURLVariables[i].split('=');

    if (sParameterName[0] === sParam) {
        return sParameterName[1] === undefined ? true : sParameterName[1];
    }
    }
};

// Given http://dummy.com/?technology=jquery&blog=jquerybyexample
var tech = getUrlParameter('technology');

// Get URL Parameter from hash
const queryString = window.location.hash.substr(1);
let urlParams = new URLSearchParams(queryString); // not supported in IE
let newToken = urlParams.get('access_token');

// Get last path
var path = window.location.pathname;
var lastindex = path.lastIndexOf('/');
if ( lastindex > 0 && path.length === lastindex + 1) {
    // remove last slash
    path = path.substring(0, path.length - 1);
    lastindex = path.lastIndexOf('/');
}
console.log(path.substring(lastindex + 1));

// Remove hash
function removeHash() { 
    history.pushState("", document.title, window.location.pathname + window.location.search);
}

8.13. Cookie js:cookie

https://developer.mozilla.org/en-US/docs/Web/API/Document/cookie 

cookieVal = "abc";
cookieKeyValPair = "litest=" + encodeURIComponent(cookieVal); 
// only one single cookie can be set/updated at a time
document.cookie = cookieKeyValPair;

// Optional cookie attribute values, preceded by a semi-colon separator
var d = new Date();
d.setTime(d.getTime() + 30*24*60*60*1000); // 30 days. If not set, cookie will be exired after browser is closed.
var expires = "expires=" + d.toUTCString();
cookieKeyValPair += "; " + expires;
cookieKeyValPair += "; path='/mydir'";

// or
cookieKeyValPair += "; path=/"

function setCookie(cname, cvalue, exdays) {
    var d = new Date();
    d.setTime(d.getTime() + (exdays*24*60*60*1000));
    var expires = "expires="+ d.toUTCString();
    document.cookie = cname + "=" + encodeURIComponent(cvalue) + ";" + expires + ";path=/";
}

8.14. Object only, no Class js:Object

8.14.1. Define an object

var key = 'hello';
var a = {key: 123}; // {key: 123}

var b = {};
b[key] = 123; // b === {hello: 123}

var c = {
    [key]: 123
};
// ES6 c === {hello: 123}

8.14.2. Contructor new, Object.create

  • Every thing in JavaScript is an object. No class
  • Every object has a prototype object
  • Example
    • Parent is an object with a constructor (because it's a function) and a prototype object
    • When new keyword is used, var parent = new Parent();, parent copies the Parent's prototype object and run the Parent's constructor
    • If the constructor returns anything, parent will take it
    • If the constructor returns nothing, the Parent object with prototype object and Parent's other properties defined in the constructor this.xyz = ... will be copied
    • copies Parent's prototype object and return it. So Parent's constructor is not run js:prototype
  • var a = Object.create(null);

  • Create an object with extra properties 

    Object.create(Object.prototype, 
{
 foo: {writable: true, configuration: true, value: 'hello', get: ..., set: ..., otherthings: ...},
 bar: {...}
})

8.14.3. Methods of Object constructor

8.14.3.1. Object.create()
8.14.3.2. Object.freeze() js:Object:freeze
const obj = { prop: 42 };

Object.freeze(obj);

obj.prop = 33; // Throws an error in strict mode

console.log(obj.prop); // expected output: 42

8.14.4. instanceof js:instanceof

  • A useful operand that is for object to test instantiation
  • myobject instanceof Myconstructor
    object
    an instance (an object)
    Myconstructor
    Function object to test against. It has to be a function name
    (no term)
    Return true if Myconstructor.prototype exists in myobject's prototype chain
  • Use instanceof to test instantiation, and use js:prototype.isPrototypeOf to test inheritance

8.14.5. Object.defineProperty js:Object.defineProperty

// To make a method non enumerable
// - `CustomerBooking` class and its method `setFilm` is already defined
Object.defineProperty(Customerbooking.prototype,'setFilm',
          {enumerable:false});

// Define a new non-enumerable key `privateField` and value `5`
Object.defineProperty(obj, 'privateField', {value: 5});

8.14.6. object.assgin Object assignment 

var anyVariable = anyObject; // pass by reference
const food = {beef: 'beef', bacon: 'bacon'}

//region Non-Deep clone an object

// Only enumerable properties are copied :: https://www.samanthaming.com/tidbits/70-3-ways-to-clone-objects/

// "Object.assign"
// mutates target object food2 so that food2 copies all enumerable own properties from 1 or more source objects.
// It also returns the target object
var food2 = {};
console.log(Object.assign(food2, food));
// This is way is preferred
var food3 = Object.assign({}, food)

// JSON, functions are replaced with null and functions in object are replaced with undefined
var food2 = JSON.parse(JSON.stringify(food));

// Spread
// Shallow copy :: the first level is copied but deeper levels are referenced
var food2 = {...food}
//endregion

8.14.7. Compare objects without methods (JSON-style)

JSON.stringify(obj1) === JSON.stringify(obj2)

8.14.8. Reuse previous value when declaring

var obj = {
  key1: "it ",
  key2: "",
  get key3() {
    console.log("key3 get is run multiple times");
    return this.key2 + " hello";
  },
  set init(dummy) {
    console.log("key2 is set only once");
    this.key2 = this.key1 + " hello";
  }
};
obj.init = "";
console.log(obj.key2 === obj.key1 + " hello");
console.log(obj.key3 === obj.key2 + " hello");
console.log(obj.key3 === obj.key2 + " hello");
obj.key3 = "override";
console.log(obj.key3 === obj.key2 + " hello");

8.14.9. Example

function CustomerBooking(bookingId, customerName, film, showDate) {
/* This is a constructor. It is not a good way to define property
 * Better to use methods to set. But here should provide initialization
*/
  this.customerName = customerName;
  this.bookingId = bookingId;
  this.showDate = showDate;
  this.film = film;
// You can also call a method which is defined later to initialize properties.
}

CustomerBooking.prototype.getCustomerName = function() {
 return this.customerName;
}

CustomerBooking.prototype.setFilm = function(film) {
 this.film = film;
}

// Instantiate
var firstBooking = new CustomerBooking(1234,
 "Arnold Palmer", "Toy Story", "27 July 2004 20:15");

var enumerated = [];
for (var prop in firstBooking) {
 enumerated[enumerated.length] = prop;
}
// ["customerName","bookingId", "showDate", "film", "getCustomerName", ..., "setFilm"]

// A cinema can hold multiple CustomerBooking instances
function cinema() {
 this.bookings = new Array();
}

cinema.prototype.addBooking = function(bookingId, customerName, film, showDate) {
 this.bookings[bookingId] = new CustomerBooking(bookingId, customerName, film, showDate);
}

// Loop through bookings of a cinema
cinema.prototype.getBookingsTable = function() {
 var booking;
 var bookingsTableHTML = "<table border=1>";
 for (booking in this.bookings) {
  bookingsTable += this.bookings[booking].getBookdingId();
  ...
 }
 bookingsTableHTML += "</table>";
 return bookingsTableHTML;
}

var londonOdeon = new cinema(); // Constructor takes no parameters
londonOdeon.addBooking(342, "First Last", "Toy Story", "15 July 2004 20:15");

8.15. Extending Native Object

Native prototype can't be deleted or replaced But values of its properties can be modified or created

Create a new method in Array class which removes a member after checking if the method is already defined 

Array.prototype.remove = Array.prototype.remove || function(member) {
 var i = this.indexOf(member);
 if (i > -1) {
  this.splice(index,1);
 } 
 return this;
}

// If a method (remove) is created in native class, but later it was assigned as a property error will show

Newly added remove method is enumerable in js:for…in js:prototype.propertyIsEnumerable Inherited properties are not enumerable To filter out method in js:for…in 

var props = [];
for (var prop in results) {
 results.hasOwnProperty(prop) && props.push(prop);
}

Add "method" in Function class Then "method" exists in any function. 

Function.prototype.method = function(name, func) {
 this.prototype[name] = func;
 return this;
}

// Parenizor is a custom class. Create a method called 'setValue'
Parenizor.method('setValue', function(v) {
 this.value = value;
 return this;
});

Add "inherits" in Function class 

Function.method('inherits', function(parent) {
 // parent is a class that is already defined
 // make a new instance
 this.prototype = new parent();
 var d = {},
     p = this.prototype;
 this.prototype.constructor = parent;
 this.method('uber', function uber(name) {
  if (!(name in d)) {
   d[name] = 0;
  } 
  var f, r, t = d[name], v = parent.prototype;
  if (t) {
   while (t) {
    v = v .contructor.prototype;
    t -= 1;
   }
   f= v[name];
  }
  else {
   f = p[name];
   if (f == this[name]) {
    f = v[name];
   }
  }
  d[name] +=1;
  r = f.apply(this, Array.prototype.slice.apply(arguments, [1]));
  d[name] -=1;
  return r;
 });
 return this;
});

8.16. Classical Inheritance, Parasitic Inheritance js:inheritance

Classical inheritance is about the is-a relationship. Parasitic inheritance is about the was-a-but-now's-a relationship.

8.16.1. Classical inheritance

Refer to: Add "method" in Function class, Add "inherits" in Function class

8.16.1.1. Example
function Parenizor(v) {
// enclose v with ( and )
 this.setValue(v);
}

Parenizor.prototype = {
  constructor: Parenizor, // This line is crucial as we are overwriting Parenizor.prototype rather than adding
  setValue: function(v) {
    this.value = v;
    return this;
  },
  getValue: function() {
    return this.value;
  },
  toString: function() {
    return '(' + this.getValue() + ')';
  }
};

myParenizor = new Parenizor(0); 
myString = myParenizor.toString(); // "(0)"

// Subclass inherits parent class Parenizor
function ZParenizor(v, w) {
 Parenizor.call(this, v); // call parent class parenizor constructor
 // You can also call other parent class constructor
 // OtherParenizor.call(this, v);

 // Define other properties that this subclass should have
 this.otherValue = w;
}

ZParenizor.prototype = Object.create(Parenizor.prototype); 
// Inherits parent class's all methods. 
// First create an instance of the parent class and assign it to the child class.

// Inherits other parent class's all methods
// mixin(ZParenizor.prototype, OtherParenizor.prototype);

ZParenizor.prototype.constructor = ZParenizor; 
// Set subclass constructor 
// but ZParenizor should have its own constructor which is ZParenizor.prototype.constructor.

// Modify subclass's inherited method toString 

ZParenizor.prototype.toString = function() {
  // ...
}

var zParenizor = new ZParenizor('v', 'w');

console.log(zParenizor instanceof ZParenizor); // true
console.log(zParenizor instanceof Parenizor); // true

8.17. OOP Pattern

  • This is not a singleton pattern. See 8.5.7
// Simple OOP pattern or prototypal pattern
Vehicle = function (year, make, model) {
    //region Public properties
    this.year = year;
    this.make = make;
    this.model = model;
    //endregion
};
Vehicle.prototype.publicProperty = '';
Vehicle.prototype.getInfo = function () {
    // Use private properties
    return this.year + ' ' + this.make + ' ' + this.model;
};
Vehicle.prototype.startEngine = function () {
    return 'Vroom';
};

(function () {
    // this === self === window
    this.myApp = this.MyApp || {};
    var ns = this.myApp; // further shorten the keyboard typing

    // private properties
    var vehicleCount = 5;
    var vehicles = [];

    // Public properties
    ns.publicHello = 'This is public';

    //region Public methods
    ns.getVehicleCount = function () {
    return vehicleCount;
    };
    ns.setVehicleCount = function (value) {
    vehicleCount = value;
    };
    //endregion

    // Define a Class
    // You can even separate the class definition into another JavaScript file
    // Just remember to include the necessary closure like the above
    ns.Vehicle = (function () {

    // Don't Do This!
    // - Since it's not in prototype, this privateStaticProperty cannot be inherited
    // - And if any prototypal method uses this variable in child class, it will throw error
    var privateStaticProperty = 'Vehicle Private Property';

    function Vehicle(year, make, model) {
        //region public properties
        this.year = year;
        this.make = make;
        this.model = model;
        //endregion
    }

    //region Public methods
    Vehicle.prototype.getInfo = function () {
        // Use public properties
        return this.year + ' ' + this.make + ' ' + this.model;
    };
    Vehicle.prototype.startEngine = function () {
        return 'Vroom';
    };
    Vehicle.prototype.setPrivateStaticProperty = function (value) {
        privateProperty = value;
    };
    Vehicle.prototype.getPrivateStaticProperty = function () {
        return privateProperty;
    };
    //endregion

    // Better not to override prototype, don't do this:
    // Vehicle.prototype = {getInfo: function () {}, startEngine: function () {}}

    return Vehicle;
    }());

    // Define a sub Class
    ns.Car = (function (parent) {

    function Car(year, make, model) {
        parent.call(this, year, make, model);
        this.wheelQuantity = 4;
    }

    // Inherit parent's prototype properties
    Car.prototype = Object.create(parent.prototype);
    // Override the constructor to this child class
    Car.prototype.constructor = Car;
    // Extend or override a parent's method
    Car.prototype.getInfo = function () {
        return 'Vehicle Type: Car ' + parent.prototype.getInfo.call(this);
    }

    return Car;
    }(ns.Vehicle));

}());

console.log(myApp.publicHello);
// undefined for private variable myApp.vehicleCount
myApp.getVehicleCount();
myApp.setVehicleCount(10);

// Instantiate
var v = new myApp.Vehicle(2012, 'Toyota', 'Rav4');
console.log(v.getInfo());

var c = new myApp.Car(2012, 'Toyota', 'Rav4');
console.log(c.getInfo());

// Instantiate Simple OOP
var vSimple = new Vehicle(2022, 'Lexus', 'UX');
console.log(vSimple.getInfo());

8.18. Promise

  • Promise is now native in all browsers including Edge except IE

  • IE polyfill https://github.com/stefanpenner/es6-promise 

    <script src="https://cdn.jsdelivr.net/npm/es6-promise@4/dist/es6-promise.auto.min.js"></script>

  • https://web.dev/promises/ 

    function get(url) {
  // You should wrap all code into return
  return new Promise(function (resolve, reject) {
// run resolve or reject function to resolve or reject a promise
var req = new XMLHttpRequest();
req.open('GET', url); // it can be set to async

req.onload = function () {
  if (req.status == 200) {
    resolve(req.response);
  }
  else {
    reject(Error(req.statusText));
  }
};

req.onerror = function () {
  reject(Error("Network Error"));
}

  });
}

get('story.json')
  .then(function (response) {
console.log('Success!', response);
// transform a value to the next .then
// just return the value, you may modify it
// The returned value has to be a non-promise
return JSON.parse(response);
  }, function (error) {
console.log('Failed!', error);
return error;
  })
  .then(function (response) {
console.log('JSON!', response);
  });

// If a 'then' just returns a value when it's resolved, use a shortform
get('story.json').then(JSON.parse).then(function(response) {
  console.log('JSON!', response);
});

// In 'then', you can return a promise
// a promise.then is a promise

function getJSON(url) {
  return get(url).then(JSON.parse);
}

getJSON('story.json').then(function(story) {
  return getJSON(story.chapterUrls[0]);
}).then(function(chapter1) {
  console.log('Got Chapter 1', chapter1);
});

// Store the result of a promise and reuse it
var storyPromise;

function getChapter(i) {
  storyPromise = storyPromise || getJSON('story.json');

  return storyPromise.then(function(story) {
return getJSON(story.chapterUrls[i]);
  });
}

getChapter(0).then(function(chapter) {
  console.log(chapter);
  return getChapter(1);
}).then(function(chapter) {
  console.log(chapter);
});

// .catch(function(e) { ... }) equals to
// .then(undefined, function(e) { ... })

// Dynamically chained .then()
// Start with a resolved promise: Promise.resolve('initialValue')
// a reject promise: Promise.reject('initialValue')
getJSON('story.json')
  .then(function(story) {
// addHTMLToPage(story.heading);
return story.chapterUrls.reduce(function(sequence, chapterUrl) {
  return sequence
    .then(function() { return getJSON(chapterUrl); })
    .then(function(chapter) { /* addHtmlToPage(chapter.html); */ });
}, Promise.resolve());
  })
  .then(function() { /* addTextToPage("All done"); */ })
  .catch(function(err) { /* addTextToPage("broken" + err.message); */ })
  .then(function() { /* document.querySelector('.spinner').style.display='none'; */ });

// Start a set of promises in parallel and create a promise that fulfils when all are fulfilled
getJSON('story.json')
  .then(function(story) {
// addHTMLToPage(story.heading);
return Promise.all(
  story.chapterUrls.map(getJSON)
); // not a callback. It's an array of Promises (not chained)
// After promises are added to array as values, they are started already

// Start a set of promises in parallel, chain them so that we get response in order

return story.chapterUrls.map(getJSON)
  .reduce(function (sequence, chapterPromise) {

    return sequence
      .then(function () {
    return chapterPromise;
      })
      .then(function (chapter) {
    addHtmlToPage(chapter.html);
      });

  }, Promise.resolve());


  })
  .then(function() { /* ... */ })
  .catch(function(err) { /* ... */ })
  .then(function() { /* ... */ });

8.19. Strict mode

  • https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Strict_mode
  • Strict mode applies to entire scripts or to individual functions

    • Function 

      function myStrictFunction() {
  // Function-level strict mode syntax
  "use strict";
  function nested() {
    return "And so am I!";
  }
  return `Hi! I'm a strict mode function! ${nested()}`;
}
    • It doesn't apply to block statements enclosed in {} braces; attempting to apply it to such contexts does nothing.
    • eval code, Function code, event handler attributes, strings passed to WindowTimers.setTimeout(), and related functions are entire scripts, and invoking strict mode in them works as expected

8.20. this and self

  • https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/this
  • It refers to an instance which is created using new keyword ora as object literals. Otherwise, this will default to the window scope
  • Arrow functions are different and follow the normal variable lookup rules
    • So while searching for this which is not present in the current scope, an arrow function ends up finding the this from its enclosing scope

    • https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions 

      window.age = 10;

function Person () {
    this.age = 42; // this Person instance (beause it is created by new Person)
    // will have property age as 42
    console.log('this.age outside setTimeout', this.age); // 42

    setTimeout(function () {
        // This is a traditional function (anonymous) and it is not created by
        // using `new` So `this` is the window scope
        console.log('this.age', this.age); // 10
    }, 100);

    setTimeout(() => {
        // Arrow function executing in the "p" (an instance of Person) scope
        console.log('this.age', this.age); // yields "42" because the function
        // executes on the Person scope
    }, 100);

  }

var p = new Person();
console.log('window.age', window.age); // 10
  • there is no self keyword in JavaScript. If JavaScript is run on a browser, window.self = window and all properties of window can be directly accessed. So self is windows unless it's overriden in other scopes

8.20.1. globalThis

  • The global this value e.g. the global object
  • Different environments have different global this object
    On the web
    window, self, frames
    (no term)
    js:worker only uses self
    Node.js
    global
    Prior to globalThis, the only reliable cross-platform way to get is
    Function('return this')()
    • Some environments that disable eval() in Function(''), like header:csp in browsers, prevent it this usage

8.20.2. this in function context

function f1() {
  return this;
}

// In a browser:
f1() === window; // true

// In NodeJS:
f1() === globalThis; // true

function f2() {
  'use strict'; // see strict mode
  return this;
}
f2() === undefined; // true

8.20.3. this in class context

  • All non-static methods within the class are added to the prototype of this
    • Static methods are properties of the class itself
class Example {
  constructor() {
    const proto = Object.getPrototypeOf(this);
    console.log(Object.getOwnPropertyNames(proto));
  }
  first(){}
  second(){}
  static third(){}
}

new Example(); // ['constructor', 'first', 'second']

8.20.4. this in a method of an object

  • The method is assigned into an instance (could be its original definition or other defintion)
  • this value is the instance's value
8.20.4.1. Example 1
abc = {
    a: 10,
    b: function () {
    return this.a;
    },
};

// traditional function (anonymous) `b` is in an instance `abc`, the instance's `a` is 10
console.log(abc.b(), 'abc.b()'); // 10
console.log(window.a, 'window.a'); // undefined

var o = { prop: 37 };

function independent () {
    return this.prop;
}

o.f = independent;
// the called traditional function `independent` is now in an instance `o`, the instance's `prop` is 37
console.log(o.f(), 'o.f()'); // 37

// the called traditional function `independent` is now in an instance `b`, the instance's `prop` is 42
o.b = { g: independent, prop: 42 };
console.log(o.b.g(), 'o.b.g()'); // 42

var o = { f: function () { return this.a + this.b; } };
var p = Object.create(o);
p.a = 1;
p.b = 4;
// the called traditional function `f` is now in an instance `p`, the instance's `a` is 1 and `b` is 4
console.log(p.f(), 'p.f()'); // 5

xyz = {
    a: 100,
    b: abc.b,
};

// abc.b now belongs to instance xyz and xyz's `a` is 100
console.log(xyz.b(), 'xyz.b()'); // 100
8.20.4.2. Example 2
function sum () {
  return this.a + this.b + this.c;
}

var o = {
  a: 1,
  b: 2,
  c: 3,
  get average () {
    return (this.a + this.b + this.c) / 3;
  },
};

Object.defineProperty(o, 'sum', {
  get: sum, enumerable: true, configurable: true,
});

console.log(o.average, o.sum); // 2, 6

8.20.5. this in a DOM event handler

my_element.addEventListener('click', function (e) {
  console.log(this.className);           // logs the className of my_element
  console.log(e.currentTarget === this); // always `true`
  console.log(this === e.target); // true when currentTarget and target are the
                  // same object
});

my_element.addEventListener('click', (e) => {
  console.log(this.className);           // WARNING: `this` is not `my_element`
  console.log(e.currentTarget === this); // `false`
});

In an inline event handler 

<button onclick="alert(this.tagName.toLowerCase());">
  <!-- alert `button` -->
  Show this
</button>

<button onclick="alert((function() { return this; })());">
  <!-- alert the window object -->
  Show inner this
</button>

<script>
  function logID() { console.log(this.id); }
</script>
<table id="my_table" onclick="logID();"><!-- when called, `this` will refer to the global window object -->
  ...
</table>

8.21. Observable js:observable

A promise is a push mechanism that calls some code after the promise has been resolved or rejected with a single value. An observable is like a promise, but it calls some code every time a new value becomes available, and can emit many values over time.

8.22. Worker

  • IE 10 and above supports worker
  • Workers are independent and separated from parent window.
  • Don't try to access or modify parent window DOM. Do it indirectly
var w;

if (typeof(Worker) !== "undefined") {
 if (typeof(w) == "undefined") {
  w = new Worker("demo_workers.js");
 }
 // Receive message from worker
 w.onmessage = function(event) {
  console.log(event.data);
 }

 w.onerror = function(error) {
  console.log(error.message, error.filename, error.lineno);
 }

 // Send message to worker
 w.postMessage(['First Name', 'Last Name']);
}

w.terminate();
w = undefined;

// demo_workers.js
var i = 0;

function timedCount() {
 i = i + 1;
 postMessage(i);
 setTimeout("timeCount()", 500);
}

close(); // Worker can close itself

// This is how worker receives a message
onmessage = function(e) {
  console.log("Message receved by worker from main window");
  var workerResult = 'Result: ' + (e.data[0] * e.data[1]);
  console.log('Posting message back to main window');
  postMessage(workerResult);
}

// Workers can spawn or create subworkers. URIs of subwokers are resolved relative to
// the parent worker's location rather than that of the owning page.

// Worker can load scripts within the same domain
// Scripts are loaded and executed in order.
importScripts('foo.js');
importScripts('foo.js','bar.js');

// Spawn a shared worker. IE doesn't support sharedWorker
// Worker Square and Worker Multiply2Numbers both use shared worker Multiply
// In sqaure.js and multiply2numbers.js

if (!!window.SharedWorker) {
 var sharedWorker = new SharedWorker("multiply.js");
 sharedWorker.port.postMessage([123,123]); // square.js
 sharedWorker.port.postMessage([123,789]); // multiply2numbers.js

 sharedWorker.port.start(); // Only needed in parent thread if onmessage is not defined
 // such as the shares worker is a callback of event listener in parent thread

 sharedWorker.port.onmessage = function(e) {
  //
  console.log(e.data); // receive from shared worker
 }
}

// In multiply.js shared worker
port.start(); // only needed if onmessage is not defined in parent thread

onconnect = function(e) {
 var port = e.ports[0];

 port.onmessage = function(e) {
  var workerResult = 'Result: ' + (e.data[0] * e.data[1]);
  port.postMessage(workerResult);
 }
}

8.23. XMLHttpRequest

8.23.1. Basics

  • Event handlers
    onload
    Always use onload instead of onreadystatechange
    • eq. to onreadystatechange and this.readyState == 4
    (no term)
    onreadystatechange
    • readyState property values
      • 0: request not initialized
      • 1: connection established
      • 2: request received
      • 3: processing request
      • 4: request finished and response is ready
    (no term)

    upload 

    rq.upload.onprogress = function (e) {
    var gId = document.getElementById;
    gId('cancel').style.display = 'block';
    if (e.lengthComputable) {
        gId('debug').innerHTML = Math.floor(e.loaded * 100 / e.total) + '%';
        if (Math.floor(e.loaded * 100 / e.total) === 100) {
            gId('debug').innerHTML = 'Processing...';
        }
    } else {
        gId('debug').innerHTML = 'uploading...';
    }
};
    (no term)

    onabort 

    rq.onabort = function () {
    gid('cancel').style.display = 'none';
    gid('debug').innerHTML = 'canceled';
}
    Others
    onloadstart, onprogress, onerror, ontimeout, onloadend
  • Methods
    • send
    • setRequestHeader
      • Specify request header Content-type manually to x-www-form-urlencoded and send('postDataInStringWith&=Separated')
      • send() with 8.23.3, Content-type is set to multipart/formdata automatically
  • this.propertyName
    readyState
    See 1.2.1
    (no term)
    responseText
    (no term)
    responseXML
    statusText
    "OK" or "Not Found"
function loadDoc() {
    var xhr = new XMLHttpRequest();

    xhr.onload = function () {
    if (this.status === 200) {
        console.log(
        this.responseText // as text
        , this.responseXML // as it's XML data
        , this.statusText // "OK" or "Not Found"
        );

        console.log(xhr.getAllResponseHeaders());
        xhr.getResponseHeader("Last-Modified");

    }
    };

    xhr.onreadystatechange = function () {
    // if (xhr.readyState == 4 && xhr.status == 200) { ... }
    }

    xhr.open('GET', 'ajax_info.txt'
    , true // async
    , "username" // optional
    , "psw" // optional
    );

    /* After open() */

    xhr.setRequestHeader("Content-type", 'application/x-www-form-urlencoded');
    // set request header. Especially if 'POST' form fields
    xhr.setRequestHeader('Accept', 'application/json, text/javascript');

    // it's common for JavaScript client to add this request header
    xhr.setRequestHeader('X-Requested-With', 'XMLHttpRequest');
    // On PHP, the request header appears as $_SERVER['HTTP_X_REQUESTED_WITH']

    // For receiving a binary string
    // xhr.overrideMimeType("text/plain; charset=x-user-defined");

    /* before send(); */

    xhr.send(); // Nothing when 'GET'
    /* For POST
     * var fname = encodeURIComponent('Henry');
     * var varJson = encodeURIComponent(JSON.stringify(var1));
     * xhr.send("fname="+fname+"&var2="+varJson);
     * if .setRequestHeader content-type is www-form-urlencoded
     * when data is received from PHP, you need to
     * json_decode(stripslashes($_POST['var2']))
     */

    // binary data can be sent
    /* var blob = new Blob(['abc123'], {type: 'text/plain'});
     * xhr.send(blob);
     * var myArray = new ArrayBuffer(512);
     * var longInt8View = new Uint8Array(myArray);
     * for (var i=0; i< longInt8View.length; i++) {
     *  longInt8View[i] = i % 255;
     * }
     * xhr.send(myArray);
     */
}

8.23.2. application/x-www-form-urlencoded vs multipart/form-data

  • Common
    • PHP will have $_POST if the request content-type is either of these 2
    • To interpret other content type or other request method e.g. 5
      • $_POST = json_decode(file_get_contents('php://input'), true);
      • $str = "first=value&arr[]=foo+bar&arr[]=baz";
  • application/x-www-form-urlencoded
    • Transfers extra data if large amount of binary data is sent
    • spaces are encoed as plus signs
  • multipart/form-data
    • HTML form submit uses js:formdata to POST multipart/form-data

    • Request header

      • Content-Type: multipart/form-data; boundary=user-agent-sets-this-boundary
      • user-agent-sets-this-boundary
      • can be any 1 to 70 characters and not ending with white space
      • characters are a set of characters known to be very robust through email gateways. Use numbers only to be safe
      • usually starts and ends with 2 dashes

    • Request Payload 

      --user-agent-sets-this-boundary
Content-Disposition: form-data; name="gv-inv_image"; filename="a.jpg"
Content-Type: image/jpeg



--user-agent-sets-this-boundary
Content-Disposition: form-data; name="description" 

some text
--user-agent-sets-this-boundary--

8.23.3. FormData to send file

  • Use JavaScript and HTML to POST multipart/form-data
  • Not for IE 9 and below

HTML to include a file upload field 

<form action="" method="post" enctype="multipart/form-data">
    <div>Pictures:
    <input type="file" name="pictures[]" />
    <input type="file" name="pictures[]" />
    <input type="file" name="pictures[]" />
    <input type="submit" value="Send" />
    </div>
    <!-- Or just one -->
    <input type="file" id="gv-inv_image">
</form>
<button onclick="upload(this);return false;">Upload</button>

var fileInputElement = document.getElementById('gv-inv_image');

var formData = new FormData();

formData.append("username", "Groucho");
formData.append("accountnum", 123456); // number 123456 is immediately converted to a string "123456"

// if value is not string or Blob or file, it will be converted to string

// HTML file input, chosen by user
formData.append("userfile", fileInputElement.files[0]);

// you can check the file before it gets uploaded

// JavaScript file-like object
var content = '<a id="a"><b id="b">hey!</b></a>'; // the body of the new file...
var blob = new Blob([content], { type: "text/xml"});

formData.append("webmasterfile", blob);

var request = new XMLHttpRequest();
request.open("POST", "http://foo.com/submitform.php");
request.send(formData);
// request content type is multipart/form-data

php 

if (isset($_FILES['gv-inv_image'])) {
  $file         = $_FILES['gv-inv_image']['name'];
  $ext          = pathinfo($file, PATHINFO_EXTENSION);
  $filename = sha1(rand(25, 1920) . rand(72, 2560)) . '-' . time();
  $filename = strtoupper($filename) . '.' . $ext;
  if (move_uploaded_file($_FILES['gv-inv_image']['tmp_name'], '../uploads/' . $filename)) {
      $result = ['status' => 'OK', 'filename' => $filename];
  }
  else {
      $result = ['status' => 'ERR', 'message' => 'File Move Error.'];
  }
}

8.23.4. Receive binary as arraybuffer or blob

var oReq = new XMLHttpRequest();
oReq.open("GET", "/myfile.png", true);
oReq.responseType = "arraybuffer";

oReq.onload = funciton(event) {
 var arrayBuffer = oReq.response; // not .responseText
 if (arrayBuffer) {
  // Read arraybuffer
  var byteArray = new Uint8Array(arrayBuffer);
  for (var i=0; i < byteArra.byteLength; i++) {
   // do something
  }
  // Read arraybuffer.

  // Construct a blob from arraybuffer
  var blob = new Blob( [arrayBuffer], {type: "image/png"} ); 

 }
};

oReq.send();

Specify the received data is blob 

var oReq = new XMLHttpRequest();
oReg.open("GET", "/myfile.png", true);
oReq.responseType = "blob";

oReq.onload = function(e) {
 var blob = oReq.response;
}

oReq.send();

8.23.5. Download a JSON file

index.php 

function hello () {
    var xhr = new XMLHttpRequest()
    xhr.onload = function () {
    if (this.status === 200) {
        var disposition = this.getResponseHeader('content-disposition')
        var matches = /"([^"]*)"/.exec(disposition)
        // if the response file does not have a name, give a default file name file.json
        var filename = (matches != null && matches[1] ? matches[1] : 'file.json')
        var blob = new Blob([this.response], { type: 'application/json' })
        var link = document.createElement('a')
        link.href = window.URL.createObjectURL(blob)
        link.download = filename
        document.body.appendChild(link)
        link.click()
        document.body.removeChild(link)
    }
    }

    // I found setting the response type is optional
    //xhr.responseType="blob";
    xhr.open('GET', 'index2.php', true)
    xhr.send()
}

index2.php 

<?php
http_response_code(200);
header('Content-Disposition: attachment; filename="filename.json"');

$result = [ 'data' => 'hello' ];

echo json_encode($result);

8.25. Apply XSLT on XML

function loadXMLDoc(filename) {
 var xhttp = new XMLHttpRequest();
 xhttp.open("GET", filename, false);
 try {
  xhttp.responseType = "msxml-document";
 }
 catch(err) {}
 xhttp.send();
 return xhttp.responseXML;
}

var xml = loadXMLDoc('cdcatalog.xml');
var xsl = loadXMLDoc('cdcatalog.xsl');

var xsltProcessor = new XSLTProcessor();
xsltProcesser.importStylesheet(xsl);
resultDocument = xsltProcesser.transformToFragment(xml, document);
document.getElementById("example").appendChild(resultDocument);

8.26. Apply XPath

var x=loadXMLDoc("books.xml");
var xml = x.responseXML;
path = "/bookstore/book[1]/title";

var nodes = xml.evaluate(path, xml, 
null, // namespaceResolver
XPathResult.ANY_TYPE, // resultType
null // If an existing XPathResult object is specified, it will be reused
); // Return xpathResult object 
var result = nodes.iterateNext();

while (result) {
 console.log(result.childNodes[0].nodeValue);
 result = nodes.iterateNext();
}

8.27. Web APIs

8.27.1. WebSocket js:ws

Javascript on client (e.g. web browser) 

var connection = new WebSocket('ws://abc.com/echo', ['soap', 'xmpp']);
// ['soap', 'xmpp'] are subprotocols

// When connection is open, send some data
connection.onopen = function() {
 connection.send('Ping'); // send string
};

// When connection close
connection.onclose = function() {
  // do something
};

// Send ArrayBuffer as binary
var img = canvas_context.getImageData(0, 0,400, 320);
var binary = new Unit8Array(img.data.length);
for (var i = 0; i < img.data.length; i++) {
 binary[i] = img.data[i];
}
connection.send(binary.buffer);

// Send file as Blob
var file = document.querySelector('input[type="file"]').files[0];
connection.send(file);

connection.onerror = function(error) {
 console.log('WebSocket Error ' + error);
};

connection.onmessage = function(e) {
 console.log(e.data);
};

// Define binary type before onmessage to receive binary
connection.binaryType = 'arraybuffer'; // or blob
connection.onmessage = function(e) {
 console.log(e.data.byteLength);
};

8.27.2. Mutation Observer

MDN MutationObserver MDN MutationRecord 

MutationObserver = window.MutationObserver;

// callback function has 2 arguments
// mutations: an array of objects of type MutationRecord
// observer: this MutationObserver instance
var observer = new MutationObserver(function (mutations, observer) {
  mutations.forEach(function (mutation) {

    // A MutationRecord has properties
    // Some of them might be different based on MutationObserver's options

    // Loop addedNodes
    // addedNodes return NodeList which doesn't have .forEach
    // js:function:call
    Array.prototype.forEach.call(
      mutation.addedNodes,
      function (addedNode) {
        if (addedNode.id == "bar") {
          console.log("bar was added!");
        }
      });
  });
});

// Define the parent element
var target = document.getElementById('foo');
// Or $('#foo').get(0)

// Which event should be observed
var config = {
  attributes: true,
  attributeOldValue: true, // set to true if attributes is set to true
  // attributeFilter: ['attr1', 'attr2'], // filter out an array of attribute local names in observation
  childList: true, // add/remove child elements including text nodes
  subtree: true, // any change to the subtree including all descendants
  characterData: true, // maybe image or multimedia changes?
  characterDataOldValue: true
};

observer.observe(target, config);

// You may stop observing later
observer.disconnect();

8.27.3. Intersection Observer js:intersection observer

Introduction by Google MDN

Supported in Edge, Firefox, Chrome, Chrome Android but not Safar and iOS

It registers a callback function which is only executed when an element enters or exists another element or intersects at a threshold and thus saves event for every scroll or viewport movement.

It doesn't tell the exact number of pixels that overlap or specifically which ones they are; however, it convers the much more common use case of "If they intersect by somewhere around N%, I need to do something". 

var io = new IntersectionObserver(
  entries => {
    console.log(entries);
  },
  {
    // Using default options: callback will be called only when the element comes partially into view and when it completely leaves the viewport. Details below
  }
);
// Start observing an element
io.observe(element);

// Stop observing an element
// io.unobserve(element);

// Disable entire IntersectionObserver
// io.disconnect();

Parameter entries are readonly IntersectionObserverEntry and they are delivered asynchronously and callback will run in the main thread.

rootBounds
DOMRect, the result of calling element.getBoundingClientRect() on the root element, which is the viewport by default.
boundingClientRect
DOMRect, the result of element.getBoundingClientRect() on the observed element.
intersectionRect
DOMRect, the intersection of the above 2 rectangles and tells which part of the observed element is visible
intersectionRatio
how much of the element is visible.

Options

root
Default null
threshold
Default [0], callback is called when intersectionRatio is greater or lower (2 times), you can set [0, 0.25, 0.5, 0.75, 1]
rootMargin
Default "0px", grow or shrink the area used for intersections.

If an iframe observes one of its elements, both scrolling the iframe as well as scrolling the window containing the iframe will trigger the callback. For the latter case, rootBounds will be set to null to avoid leaking data across origins.

8.27.5. Beacon API

8.27.6. Fetch API

https://caniuse.com/fetch
No IE
(no term)

CORS 

fetch('https://example.com', {
  mode: 'cors',
  credentials: 'include' // send cookies
})
credentials
optional. d
(no term)

No timeout built-in 

function fetchTimeout(url, init, timeout = 3000) {
return new Promise((resolve, reject) => {
    fetch(url, init)
    .then(resolve)
    .catch(reject);
    setTimeout(reject, timeout);
})
}

// Or

Promise
.race([
    fetch('http://url', {method: 'GET'}),
    new Promise(resolve => setTimeout(resolve, 3000))
])
.then(response => console.log(response))
(no term)

Force to abort. Have to use AbortController API 

const controller = new AbortController();

fetch(
  'http://domain/service',
  {
method: 'GET'
signal: controller.signal
  })
  .then( response => response.json() )
  .then( json => console.log(json) )
  .catch( error => console.error('Error:', error) );
(no term)
No progress e.g. 30% download complete
(no term)
Cookies are not sent

8.28. Testing

8.28.1. Unit Testing

QUnit

8.28.2. Performance 

var t0 = performance.now();
doSomething();
var t1 = performance.now();
console.log("Call to doSomething took " + (t1 - t0) + " milliseconds.");

// Or this way for console only
console.time('doSomething');
doSomething();
console.timeEnd('doSomething');

// On console, `console.timeEnd` prints out one line:
// doSomething: 3.259765625 ms

8.29. Worker

8.29.1. Web Worker

  • Workers are in external files which don't have access to DOM (objects like window, document and parent) and completely separate from the webpage
  • Use a Web Worker for heavy computation on the client. Use a Service Worker for offline capabilities like intercepting network requests, as well as other fun stuff like push notifications and synchronization of content in the background.
  • Web workers have many instances per tab while Service Worker has one for all tabs
  • Web workers are terminated when the tab is closed
    • Service worker can still receive messages when the tabs are closed but browsers are up
      • In Android, service workers can still receive messages when browser apps are closed
  • Service workers are also able to postMessage
  • Both are completely async with main thread
// Place to create a web worker main.js
var w;
if (typeof(Worker) !== "undefined") {
    if (typeof(w) == "undefined") {
    w = new Worker("demo_workers.js");
    }
    w.onmessage = function(e) {
    // handle message sent from web worker
    console.log(e.data);
    }
    // post message to web worker
    w.postMessage('hi');

    // manually terminate a web worker
    w.terminate();
}
else {
    // Worker not supported
}

// demo_workers.js
var i=0;
function timedCount() {
    i= i +1;
    postMessage(i);
    // post message from worker to the birth place
    setTimeout("timedCount()", 500);
}
timedCount();
onmessage = function(oEvent) {
    // receive messages sent from the birth place
    postMessage('Hi ' + oEvent.data);
};

8.29.2. Service Worker js:serviceworker

8.29.2.1. Basics
  • https://developers.google.com/web/fundamentals/primers/service-workers/
  • https://developers.google.com/web/ilt/pwa/introduction-to-service-worker
  • https://jakearchibald.github.io/isserviceworkerready/ IE11 is not supported
  • HTTPS or localhost
  • sw can/is
    • cache files so that the website can work offline
    • JavaScript Web Worker
    • ES6
    • 100% based on Promise
    • Notifications API and Push API
    • Background Sync API
    • Channel Messaging API
    • geofencing
  • sw can't
    • use localStorage
    • access DOM directly
  • Life Cycle

    • Registration 

        if ('serviceWorker' in navigator) {
window.addEventListener('load', function() {
  navigator.serviceWorker.register('/sw.js').then(function(registration) {
    // Registration was successful
    console.log('ServiceWorker registration successful with scope: ', registration.scope);
  }, function(err) {
    // registration failed :(
    console.log('ServiceWorker registration failed: ', err);
  });
});
  }
      • register() can be called many times without concern
      • Path can be relative path e.g. './sw.js', for event fetch, urls can also be relative

      • Say project is /train/* and /train/sw.js, specify URLs to resource with different query string in mind 

        register('./sw.js')

urlsToCache = [
  './', // /train/
  'index.html', // /train/index.html
  'images/a.jpg',  // /train/images/a.jpg
  'scripts/a.js'
]
      • sw receives fetch events for everything on this domain
      • sw only sees and receives fetch events for pages whose URL starts with /ex/, /ex/page1/, /ex/page2/ etc
        chrome://inspect/#service-workers
        shows an sw exists https://yoursite.com/serviceworker with a pid which you can inspect and terminiate. Incognito kills all sw if Incognitor is closed

    • Installation In sw.js 

      var CACHE_NAME = 'my-site-cache-v1';
var urlsToCache = [
'/',
'/styles/main.css',
'/script/main.js'
];

self.addEventListener('install', function(event) {
// Perform install steps
event.waitUntil(
    caches.open(CACHE_NAME)
    .then(function(cache) {
        console.log('Opened cache');

        // load other resources e.g. big resources that are not absolutely needed in first load
        // sw may be killed and unfinished downloading in this part may be dropped.
        // cache.addAll(...);

        // all resources should be downloaded otherwise this error throws and the install process fails
        // Uncaught (in promise) TypeError: Request failed

        return cache.addAll(urlsToCache);
        // Accept all URLs to fetch from and add response to the cache
        // if any files fail to download, the install step will fail
        // When all fetch promises are resolved, event activate event is triggered. 

    })
);
return self.clients.claim(); // activate sw faster
});

      // when user navigates to a different page or refreshes, sw receives fetch events
// return cache as response if cache has it, otherwise make a network request to fetch
self.addEventListener('fetch', function(event) {
event.respondWith(
    // find any cached results from any of the caches sw created.
    caches.match(event.request)
    .then(function(response) {
        // Cache hit - return response (cache value)
        if (response) {
        return response;
        }
        // fetch over the network
        return fetch(event.request);
    }
     )
);
});
    • Activation
  • event.waitUntil adds 1 promise to the install and activate event callbacks. Multiple event.waitUntil is possible inside the event callback and all promises have to settle
  • Event callbacks for install and activate have to be finished before event fetch can be run

  • return if cache is found, otherwise make a network request to fetch, check response, if not successful or not coming from the same origin, just return the response, otherwise cache that response 

    self.addEventListener('fetch', function(event) {
  event.respondWith(
caches.match(event.request)
  .then(function(response) {
    // Cache hit - return response
    if (response) {
      return response;
    }

    // IMPORTANT: Clone the request. A request is a stream and
    // can only be consumed once. Since we are consuming this
    // once by cache and once by the browser for fetch, we need
    // to clone the response.
    var fetchRequest = event.request.clone();

    return fetch(fetchRequest).then(
      function(response) {
    // Check if we received a valid response
    // 'basic' means it's a request from the current origin
    if(!response || response.status !== 200 || response.type !== 'basic') {
      return response;
    }

    // IMPORTANT: Clone the response. A response is a stream
    // and because we want the browser to consume the response
    // as well as the cache consuming the response, we need
    // to clone it so we have two streams.
    var responseToCache = response.clone();

    caches.open(CACHE_NAME)
      .then(function(cache) {
        cache.put(event.request, responseToCache);
      });

    return response;
      }
    );
  })
  );
});

    • fetch: always serve cache first, but always update cache 

      self.addEventListener('fetch', function(event) {
  event.respondWith(
    caches.open('mysite-dynamic').then(function(cache) {
  return cache.match(event.request).then(function(response) {
    var fetchPromise = fetch(event.request).then(function(networkResponse) {
      cache.put(event.request, networkResponse.clone());
      return networkResponse;
    })
    return response || fetchPromise;
  })
    })
  );
});
  • Update Service Worker (sw.js)
    • Update your service worker JavaScript file. When the user navigates to your site, the browser tries to redownload the script file that defined the service worker in the background. If there is even a byte's difference in the service worker file compared to what it currently has, it considers it new
    • Your new service worker will be started and the install event will be fired
    • At this point the old service worker is still controlling the current pages so the new service worker will enter a waiting state
    • When the currently open pages of your site are closed, the old service worker will be killed and the new service worker will take control
    • Once your new service worker takes control, its activate event will be fired
  • activate event is fired when the service worker starts up. Long event activate may block page loads. Keep it as lean as possible, only use it for things you couldn't do while the old version was active

    • Create 'pages-cache-v1' and 'blog-posts-cache-v1' and delete old 'my-site-cache-v1' 

        self.addEventListener('activate', function(event) {

var cacheWhitelist = ['pages-cache-v1', 'blog-posts-cache-v1'];

event.waitUntil(
  caches.keys().then(function(cacheNames) {
    return Promise.all(
      cacheNames.map(function(cacheName) {
    if (cacheWhitelist.indexOf(cacheName) === -1) {
      return caches.delete(cacheName);
    }
      })
    );
  })
);
  });
  • To manually refresh sw.js on DevTools, Application > Service Workers > skipWaiting, or on the top to enable Update on reload
8.29.2.3. Caching strategies

https://jakearchibald.com/2014/offline-cookbook/

Cache when an action is triggered. Caches API is available from regular pages not only sw.js 

document.querySelector('.cache-article')
.addEventListener('click', function(event) {
  event.preventDefault();

  var id = this.dataset.articleId;
  caches.open('mysite-article-' + id).then(function(cache) {
    fetch('/get-article-urls?id=' + id).then(function(response) {
  // /get-article-urls returns a JSON-encoded array of
  // resource URLs that a given article depends on
  return response.json();
    }).then(function(urls) {
  cache.addAll(urls);
    });
  });
});

Serve a offline page if both cache and network are not available 

self.addEventListener('fetch', function(event) {
  event.respondWith(
    // Try the cache
    caches.match(event.request).then(function(response) {
  // Fall back to network
  return response || fetch(event.request);
    }).catch(function() {
  // If both fail, show a generic fallback:
  return caches.match('/offline.html');
  // However, in reality you'd have many different
  // fallbacks, depending on URL & headers.
  // Eg, a fallback silhouette image for avatars.
    })
  );
});

Sample 

self.addEventListener('fetch', function(event) {
  // Parse the URL:
  var requestURL = new URL(event.request.url);

  // Handle requests to a particular host specifically
  if (requestURL.hostname == 'api.example.com') {
    event.respondWith(/* some combination of patterns */);
    return;
  }
  // Routing for local URLs
  if (requestURL.origin == location.origin) {
    // Handle article URLs
    if (/^\/article\//.test(requestURL.pathname)) {
  event.respondWith(/* some other combination of patterns */);
  return;
    }
    if (/\.webp$/.test(requestURL.pathname)) {
  event.respondWith(/* some other combination of patterns */);
  return;
    }
    if (request.method == 'POST') {
  event.respondWith(/* some other combination of patterns */);
  return;
    }
    if (/cheese/.test(requestURL.pathname)) {
  event.respondWith(
    new Response("Flagrant cheese error", {
      status: 512
    })
  );
  return;
    }
  }

  // A sensible default pattern
  event.respondWith(
    caches.match(event.request).then(function(response) {
  return response || fetch(event.request);
    })
  );
});
8.29.2.4. Push (not yet available on Chrome)

Update cache before showing a notification 

self.addEventListener('push', function(event) {
  if (event.data.text() == 'new-email') {
    event.waitUntil(
  caches.open('mysite-dynamic').then(function(cache) {
    return fetch('/inbox.json').then(function(response) {
      cache.put('/inbox.json', response.clone());
      return response.json();
    });
  }).then(function(emails) {
    registration.showNotification("New email", {
      body: "From " + emails[0].from.name
      tag: "new-email"
    });
  })
    );
  }
});

self.addEventListener('notificationclick', function(event) {
  if (event.notification.tag == 'new-email') {
    // Assume that all of the resources needed to render
    // /inbox/ have previously been cached, e.g. as part
    // of the install handler.
    new WindowClient('/inbox/');
  }
});
8.29.2.5. Background sync (not yet available on Chrome)
self.addEventListener('sync', function(event) {
  if (event.id == 'update-leaderboard') {
    event.waitUntil(
  caches.open('mygame-dynamic').then(function(cache) {
    return cache.add('/leaderboard.json');
  })
    );
  }
});

8.30. iFrame

8.30.1. Communication Between Children Window and Parent Window

  • A parent window can have multiple iframes and iframes are loaded from other domain

#+NAME Post message from iFrame to parent Window 

// In iframe
parent.postMessage('the message','*');
// or window.top.postMessage

// In parent window
function receiveMessage(event) {
 if (event.origin !== 'http://otherdomain.com') return;
 console.log(event.data);
}
window.addEventListener("message", receiveMessage, false);

#+NAME Post message from parent window to iFrame 

// In parent window
window.onload = function() {
  var receiver = document.getElementById("iFrameID").contentWindow;
  receiver.postMessage('Hello', 'http://iframe.src.domain'); // '*' for any domain of any receiver
}

// In iFrame
window.onload = function() {
  function receiveMessage(e) {
    if (e.origin !== "http://parent.window.domain") return;
    alert(e.data);
  }
  window.addEventListener('message', receiveMessage);
}

8.30.2. Dynamic iframe height

This code requires the iframe domain is same origin. 

<iframe id="preview_iframe" style="width:100%"
        onload="resize_iframe(this)"
        srcdoc="<?php echo str_replace('"', '&quot;', $html);?>"
>
</iframe>
<script>
function resize_iframe(iframe) {
  iframe.height= iframe.contentWindow.document.body.scrollHeight + 'px';
}
</script>

8.30.3. Responsive iframe

Refer to bootstrap:responsive helpers

8.31. setTimeout recall

function runTimeout() {
  setTimeout(function() {
    if (window.googletag && googletag.apiReady) {
  // do something
    }
    else {
  //console.log('googletag is not ready');
  runTimeout();
    }
  }, 5000);
}

8.32. Device Detection

var isSafari = /^((?!chrome|android).)*safari/i.test(navigator.userAgent);
console.log("Your screen resolution is: " + screen.width + "x" + screen.height);
console.log("Your screen resolution is: " + window.screen.width * window.devicePixelRatio + "x" + window.screen.height * window.devicePixelRatio);

8.33. CSS Media Viewport Size js:viewport size

  • var w = Math.max(window.innerWidth || 0, document.body.clientWidth)
  • var h = Math.max(window.innerHeight || 0, document.body.clientHeight)
  • width including scrollbars
  • width excluding scrollbars

8.34. ECMAScript

  • https://tc39.es/ecma262/
  • 1999 or 2011. Fully supported since Chrome 41
  • 6 stands for 6th edition. 2015 is the year js:es6
  • ES7-2016, ES8-2017, ES9-2018, ES10-2019

8.34.1. Arrow function

(a) => {
    return a + 100;
}

(a) => a + 100;

a => a + 100;

a => {
    return a + 100;

}

(a, b) => {
    return a + b;
}

(a, b) => a + b;

(a, b, ...r) => expression
(a = 400, b = 20, c) => expression
([a, b] = [10, 20]) => expression
({ a, b } = { a: 10, b: 20 }) => expression

async param => expression
async (param1, param2, ...paramN) => {
  statements
}
8.34.1.1. Double Arrow Functions (Currying)
// A simple arrow function
const myFunction = (x, y) => {
  console.log('param1', x);
  console.log('param2', y);
}

// The same function in curried form or as you say, double arrow function
const myFunction = x => y => {
   console.log('param1', x);
   console.log('param2', y);
}

// The above function without using arrow functions - 
function myFunction(x) {
    return function innerFunction(y) {
       console.log('param1', x);
       console.log('param2', y);
    }
}

8.34.2. Map - New Data Type (ES6)

var course = new Map();
course.set('react', {description: 'ui'});
course.set('jest', {description: 'testing'});

console.log(course); // course.size === course.length === 2
console.log(course.react); // error
console.log(course.get('react')); // an object {description: 'ui'}

// details is equivalent to course
var details = new Map([
    ['react', {description: 'ui'}], // key/value can be any data type: [new Date(), 'today'],
    ['jest', {description: 'testing'}]// key/value can be any data type: ['items', [1, 2]]
]);

console.log(details.size);

details.forEach(function(item) {
    console.log(item);
});

8.34.3. Set - New Data Type (ES6)

var books = new Set();
books.add('Pride and Prejudice');
books.add('War and Peace')
    .add('Oliver Twist');

console.log(books);
console.log('how many books?', books.size);
console.log('has Oliver Twist?', books.has('Oliver Twist'));
books.delete('Oliver Twist');
console.log('has Oliver Twist still?', books.has('Oliver Twist'));

var data = [4,2,4,4,2,5,1,6,7,5,6,8,2,7];
var set = new Set(data); // auto dedup
console.log('data.length', data.length);
console.log('set.size', set.size);

8.34.4. Generator Function

  • Treat each yield is a stop, and later code could be started when gen.next() is called 

    function* eachItem(arr) {
for(var i=0; i< arr.length; i++) {
    yield arr[i];
}
}

var letters = eachItem(["a", "b", "c", "d", "e", "f", "g"]);

var abcs = setInterval(function(){
var letter = letters.next();
if(letter.done) {
    clearInterval(abcs);
    console.log("Now I know my ABC's");
} else {
    console.log(letter.value);
}
},
           500);
  • Assign variable = yield (async requests) and program like they are normal sync. Refer to last example

  • Refer to nodejs:co for yielding promises 

    function* idMaker() {
    var index = 0;
    while (index <3)
    yield index++;
}

var gen = idMaker();
// Generator function is not contructable
// don't use var gen = new idMaker;
// When GF is called, an iterator object is returned and 
// the body is not executed yet.

console.log(gen.next().value);
// when the iterator's next() is called for the first time, 
// the GF body is executed until the 1st yield expression
// when it's called the nth time, the nth yield expression is called.

// Say there are 3 yields in total, when gen.next() is called the 4th time or more
// it will be {value: undefined, done: true}

// Use GF in another GF
function* anotherGenerator(i) {
    var index = 0;
    while (index < 3) {
    index++;
    yield i + index;
    }
}
function* generator(i) {
    yield i;
    yield* anotherGenerator(i);
    yield i + 10;
}
var gen2 = generator(10);
console.log(gen2.next().value); // ...
// 10, 11, 12, 13, 20, done

    Passing variable in next(). The variable passed will subsitute the whole yield statement for the previous iteration. (after the previous yield, and before the next/current yield) 

    function* consumer(){
    while (true){
    var val = yield null;
    console.log('Got value', val);
    }
}

var c = consumer();
c.next(1)
// No 'Got value'
c.next(2)
// Got value 2

    Usage Without GF, you will write 

    fs.readFile('blog_post_template.html', function(err, tpContent){
    fs.readFile('my_blog_post.md', function(err, mdContent){
    console.log(tpContent, mdContent);
    });
});

    With GF 

    function readFile(filepath) {
    return function(callback) {
    fs.readFile(filepath, callback);
    }
}

function run(genfun) {
    var gen = genfun();

    function next(err, answer) {
    var res;
    if (err) {
        return gen.throw(err);
    }
    else {
        res = gen.next(answer);
    }

    if (!res.done) {
        res.value(next);
    }
    }

    next();

}

run(function* () {
    try{
    var tpContent = yield readFile('blog_post_template.html');
    var mdContent = yield readFile('my_blog_post.md');
    console.log(tpContent, mdContent);
    }catch(e){
    console.error(e.message);
    }
});

8.34.5. async function

  • https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
  • async function returns AsyncFunction object which is Promise
  • await expressions make promise-returning functions behave as though they're synchronous by suspending execution until the returned promise is resolved.
    • Deals with one and only one Promise
    • Within regular JavaScript code, await is only valid inside async functions. await can be used on its own with JavaScript modules.
  • To wait for all promises to finish, use Promise.all

  • The purpose of async / await is to simplify the syntax necessary to consume promise-based APIs. The behavior of async / await is similar to combining generators and promises. 

    function resolveAfter2Seconds(x) {
  return new Promise(resolve => {
    setTimeout(() => {
      resolve(x);
    }, 2000);
  });
}

async function add1(x) {
  // start and await Promise in order
  const a = await resolveAfter2Seconds(20);
  const b = await resolveAfter2Seconds(30);
  return x + a + b;
}

add1(10).then(v => {
  console.log(v);  // prints 60 after 4 seconds.
});

async function add2(x) {
  // start Promise all together
  const p_a = resolveAfter2Seconds(20);
  const p_b = resolveAfter2Seconds(30);
  // await each Promise
  return x + await p_a + await p_b;
}

add2(10).then(v => {
  console.log(v);  // prints 60 after 2 seconds.
});

8.34.6. js:for…of

8.34.7. Import, export

8.34.7.1. ES5
// App.js
var React = require('react');
var Component = React.Component;
require('./App.css');
var Color = require('./Shapes'); // color component

class App extends Component {
  render() {
return (
  <div className="App">
    <div className="colors">
      <Color name="red"/>
      <Color name="green"/>
      <Color name="blue"/>
    </div>
  </div>
);
  }
}

module.exports = App;

// Shapes.js
var React = require('react');
var Component = React.Component;
class Color extends Component {
  render() {
const divStyle = {
  backgroundColor: this.props.name,
  color: 'white',
  fontSize: '20px',
  height: '100px',
  width: '100px'
}
return (
  <div style={divStyle}>{this.props.name}</div>
)
  }
}
module.exports = Color;


// Export multiple components
class Animal extends Component {
  render() {
const divStyle = {
  fontSize: '20px',
  height: '100px',
  width: '100px',
  border: '1px solid black',
  borderRadius: '50%'
}
return (
  <div style={divStyle}>{this.props.name}</div>
)
  }
}

module.exports = {
  Color: Color,
  Animal: Animal
}

// Multiple exports, the import needs to change, too

// App.js
var {Color} = require('./Shapes');
var {Animal} = require('./Shapes');
8.34.7.2. ES6
//App.js
import React, {Component} from 'react';

// React is the default exported from react (react.default) but Component is react.Component

import './App.css';

// Color is Shapes.Color not Shapes.default. Same for Animal
import {Color, Animal} from './Shapes';
// ...

// this way (called default import) is faster
import Button from '@mui/material/Button';

// than (called named import)
import { Button } from '@mui/material'; // this is slow


export default App;

//Shapes.js
import React, {Component} from 'react';
export class Color extends Component {
//...
}
export class Animal extends Component {
//...
}

8.34.8. Trailing commas

  • Safe in
    • array
    • object literals since ES5
    • function parameters since ES8-2017
  • Disallow in
    • JSON

8.35. TypeScript

Playground
https://www.typescriptlang.org/play/index.html
(no term)

Basics 

npm install -g typescript

tsc --version

# compile a file to greeter.js
tsc greeter.ts

# when file path is defined, tsconfig.json is not used by default
# change target
tsc --target es5 greeter.ts

8.35.1. tsconfig.json

# to create a tsconfig.json
tsc --init

# to compile based on tsconfig.json
tsc -p .

# to compile other project
tsc -p other/tsconfig.json

8.35.2. Gulp

npm init
npm install -g gulp-cli
npm i -D typescript gulp@4.0.0 gulp-typescript

# add files
gulp
node dist/main.js

#+NAME ./gulpfile.js 

var gulp = require('gulp');
var ts = require('gulp-typescript');
var tsProject = ts.createProject('tsconfig.json');

gulp.task('default', function () {
    return tsProject.src()
    .pipe(tsProject())
    .js.pipe(gulp.dest('dist'));
});

#+NAME src/greet.ts 

export function sayHello(name: string) {
    return `Hello from ${name}`;
}

#+NAME src/main.ts 

import { sayHello } from './greet';

console.log(sayHello('TypeScript'));

#+NAME ./tsconfig.json 

{
    "files": [
    "src/main.ts",
    "src/greet.ts"
    ],
    "compilerOptions": {
    "noImplicitAny": true,
    "target": "es5"
    }
}
With Browserify
npm i -D browserify tsify vinyl-source-stream
tsify is a Browserify plugin
gives access to TypeScript compiler
viny-source-stream
adapt the file output of Browserify to Gulp format called vinyl
(no term)

Example #+NAME ./src/index.html 

<!DOCTYPE html>
<html>
<head>
    <meta charset="UTF-8" />
    <title>Hello World!</title>
</head>
<body>
    <p id="greeting">Loading ...</p>
    <script src="bundle.js"></script>
</body>
</html>

#+NAME ./src/main.ts 

import { sayHello } from './greet';

function showHello(divName: string, name: string) {
const elt = document.getElementById(divName);
elt.innerText = sayHello(name);
}

showHello('greeting', 'TypeScript');

#+NAME gulpfile.js 

var gulp = require('gulp');
var browserify = require('browserify');
var source = require('vinyl-source-stream');
var tsify = require('tsify');
var paths = {
pages: ['src/*.html']
};

gulp.task('copy-html', function () {
return gulp.src(paths.pages)
    .pipe(gulp.dest('dist'));
});

gulp.task('default', gulp.series(gulp.parallel('copy-html'), function () {
return browserify({
    basedir: '.',
    debug: true,
    entries: ['src/main.ts'],
    cache: {},
    packageCache: {}
})
.plugin(tsify)
.bundle()
.pipe(source('bundle.js'))
.pipe(gulp.dest('dist'));
}));
(no term)

With Watchify npm i -D watchify fancy-log #+NAME ./gulpfile.js 

var gulp = require('gulp');
var browserify = require('browserify');
var source = require('vinyl-source-stream');
var watchify = require('watchify');
var tsify = require('tsify');
var fancy_log = require('fancy-log');
var paths = {
pages: ['src/*.html']
};

var watchedBrowserify = watchify(browserify({
basedir: '.',
debug: true,
entries: ['src/main.ts'],
cache: {},
packageCache: {}
}).plugin(tsify));

gulp.task('copy-html', function () {
return gulp.src(paths.pages)
    .pipe(gulp.dest('dist'));
});

function bundle() {
return watchedBrowserify
    .bundle()
    .on('error', fancy_log)
    .pipe(source('bundle.js'))
    .pipe(gulp.dest('dist'));
}

gulp.task('default', gulp.series(gulp.parallel('copy-html'), bundle));
watchedBrowserify.on('update', bundle);
watchedBrowserify.on('log', fancy_log);
(no term)

With Uglify npm i -D gulp-uglify vinyl-buffer gulp-sourcemaps #+NAME ./gulpfile.js 

var gulp = require('gulp');
var browserify = require('browserify');
var source = require('vinyl-source-stream');
var tsify = require('tsify');
var uglify = require('gulp-uglify');
var sourcemaps = require('gulp-sourcemaps');
var buffer = require('vinyl-buffer');
var paths = {
pages: ['src/*.html']
};

gulp.task('copy-html', function () {
return gulp.src(paths.pages)
    .pipe(gulp.dest('dist'));
});

gulp.task('default', gulp.series(gulp.parallel('copy-html'), function () {
return browserify({
    basedir: '.',
    debug: true,
    entries: ['src/main.ts'],
    cache: {},
    packageCache: {}
})
.plugin(tsify)
.bundle()
.pipe(source('bundle.js'))
.pipe(buffer()) // new
.pipe(sourcemaps.init({loadMaps: true})) // new
.pipe(uglify()) // new
.pipe(sourcemaps.write('./')) // new
.pipe(gulp.dest('dist'));
}));
(no term)

With Babel npm i -D babelify@8 babel-core babel-preset-es2015 vinyl-buffer gulp-sourcemaps Change TypeScript to output ES2015 as target. Later Babel will produce ES5 from it. #+NAME tsconfig.json 

{
"files": [
    "src/main.ts"
],
"compilerOptions": {
    "noImplicitAny": true,
    "target": "es2015"
}
}

#+NAME ./gulpfile.js 

var gulp = require('gulp');
var browserify = require('browserify');
var source = require('vinyl-source-stream');
var tsify = require('tsify');
var sourcemaps = require('gulp-sourcemaps');
var buffer = require('vinyl-buffer');
var paths = {
pages: ['src/*.html']
};

gulp.task('copy-html', function () {
return gulp.src(paths.pages)
    .pipe(gulp.dest('dist'));
});

gulp.task('default', gulp.series(gulp.parallel('copy-html'), function () {
return browserify({
    basedir: '.',
    debug: true,
    entries: ['src/main.ts'],
    cache: {},
    packageCache: {}
})
.plugin(tsify) // output is files of es2015 standard
.transform('babelify', {
    presets: ['es2015'],
    extensions: ['.ts']
}) // new. Modify Babelify setting: add .ts extension for file processing
.bundle()
.pipe(source('bundle.js'))
.pipe(buffer()) // new
.pipe(sourcemaps.init({loadMaps: true})) // new
.pipe(sourcemaps.write('./')) // new
.pipe(gulp.dest('dist'));
}));

8.36. Use Cases

8.36.1. Click to anchor

Changes the hash in URL and then go to the element 

<div onclick="goToAnchor('gotoanchor-section-name')">
  something here
</div>

<div class="spacer" id="gotoanchor-section-name"></div>

<div> Content I want to go to </div>

<script>
  function goToAnchor(anchor) {
    var loc = document.location.toString().split('#')[0];
    document.location = loc + '#' + anchor;
    return false;
  }
</script>

8.36.2. Get radio value, Event on radio buttons

<input type="radio" name="genderS" value="1" checked onclick="console.log('value is 1');"> Male
<input type="radio" name="genderS" value="0" onclick="console.log('value is 0');"> Female

// Returns the checked radio value, if no radio is checked
var checkedRadio = document.querySelector('input[name="my_radio_name"]:checked');

if (checkedRadio) {
    console.log(checkedRadio.value);
}
else {
    console.log('No radio is checked');
}

// Old
function getRadioValue(name) {
  var radios = document.getElementsByName(name);
  if (!!radios && radios.length > 0) {
    for (var i = 0, length = radios.length; i < length; i++) {
  if (radios[i].checked) {
    return radios[i].value; // only one radio can be logically checked, don't check the rest
  }
    }
  }
  return null;
}

getRadioValue('genderS');

8.36.3. Detect iOS version

https://gist.github.com/Craga89/2829457 

/*
 * Outputs a float representing the iOS version if user is using an iOS browser i.e. iPhone, iPad
 * Possible values include:
 *      3 - v3.0
 *      4.0 - v4.0
 *      4.14 - v4.1.4
 *      false - Not iOS
 */

var iOS = parseFloat(
    ('' + (/CPU.*OS ([0-9_]{1,5})|(CPU like).*AppleWebKit.*Mobile/i.exec(navigator.userAgent) || [0,''])[1])
    .replace('undefined', '3_2').replace('_', '.').replace('_', '')
) || false;

9. JavaScript projects

9.1. Vue.js

9.1.1. Basics

  • Who use it?
  • Builds

  • Example An instance is created also called viewmodel in new Vue() 

    <script src="https://unpkg.com/vue"></script>

<div id="app">
  <!-- bind attribute to a var -->
  <span v-bind:title="msg2">
{{ message }}
  </span>
  <div>
<span v-if="seen">Now you see me</span>
  </div>
  <div>
<ol>
  <li v-for="todo in todos">
    {{ todo.text }}
  </li>
</ol>
  </div>

  <button v-on:click="reverseMessage">Reverse Message</button>

  <!-- 2 way binding -->
  <div>
<input v-model="message">
  </div>
</div>

<script>
  var my_vue_app = new Vue({
el: '#app',
data: {
  message: 'Hello Vue.js!',
  msg2: 'span title',
  seen: true,
  // my_vue_app.todos.push({ text: 'New item' })
  todos: [
    { text: 'Learn JavaScript' },
    { text: 'Learn Vue' },
    { text: 'Build something awesome' }
  ]
},
methods: {
  reverseMessage: function () {
    this.message = this.message.split('').reverse().join('')
  }
}
  })
</script>

9.1.2. Data and methods vm.data vue:data

  • Root-level reactive properties need to be defined when the instance was first created to ensure reactive behavior (trigger view updates) 

    var data = {a:1}
var vm = new Vue({
  data: data
});

vm.a === data.a // true, they refer to the same object
vm.a = 2 // then data.a is 2, too
data.a = 3 // and vice-versa vm.a === 3

vm.b = 'hi' // the properties in data are active only when the instance was first created
// a new property is defined after initialization, so it won't trigger view update
// Define all variables with default values you need in initialization

  • Special instance properties and methods 

    var data = { a: 1 }
var vm = new Vue({
  el: '#example',
  data: data
})
vm.$data === data // => true
vm.$el === document.getElementById('example') // => true
// $watch is an instance method
vm.$watch('a', function (newValue, oldValue) {
  // This callback will be called when `vm.a` changes
})
  • Some plugins allow components to access/change specific root-level properties e.g. this.$store vue:plugin:vuex, this.$router vue:router
  • Don't use arrow function
9.1.2.1. Mutation, Vue.set vue:data:mutation
Mutation methods
when these methods are called, the views will be updated
  • push, pop, shift, unshift, splice, sort, reverse
  • e.g. my_vue_app.items.push({ message: 'Baz'});
(no term)

Non-mutating methods always return a new array e.g. filter, concat 

example1.items = example1.items.filter(function (item) {
  return item.message.match(/Foo/)
})
Add new prop reactivitely

Don't use vm.items[indexOfItem] = newValue, instead use: 

// Vue.set
Vue.set(example1.items, indexOfItem, newValue)

// Or

// Array.prototype.splice
example1.items.splice(indexOfItem, 1, newValue)

Don't use vm.items.length = newLenth, instead use: 

example1.items.splice(newLength)

Add reactive properties to an object at root-level 

var vm = new Vue({
  data: {
userProfile: {
  name: 'Anika'
}
  }
})

Vue.set(vm.userProfile, 'age', 27);

// or this.$set(this.userProfile, 'age', 27)
// or vm.$set

The other way to add a new prop is to rebuild the object using Object.assign() or _.extend() 

/* Don't do it directly
Object.assign(this.userProfile, {
  age: 27,
  favoriteColor: 'Vue Green'
})
*/

this.userProfile = Object.assign({}, this.userProfile, {
  age: 27,
  favoriteColor: 'Vue Green'
})

vue:plugin:vuex:mutations

9.1.2.2. vm.methods 
var vm = new Vue({
  data: { a: 1 },
  methods: {
    plus: function () {
  this.a++
    }
  }
})
vm.plus()
vm.a // 2

9.1.3. Instance Lifecycle Hooks

  • Data observation > compile templates > mount instance to DOM > Update DOM when data changes
  • Don't use arrow function in those hooks e.g. created: () => console.log(this.a)
    • this is ok mounted() => {}
  • Full lifecycle diagram and hooks
9.1.3.1. beforeCreate, created

run code after an instance is created 

new Vue({
  data: {
    a: 1
  },
  created: function () {
    // `this` points to the vm instance
    console.log('a is: ' + this.a)
  }
})
// => "a is: 1"
9.1.3.2. vm.$mount(el), beforeMount, mounted
9.1.3.3. beforeUpdate, updated
9.1.3.4. beforeDestroy, destroyed

9.1.4. Component

9.1.4.1. Basics

  • Example 

    <div id="app-7">
  <ol>
<!--
  Now we provide each todo-item with the todo object
  it's representing, so that its content can be dynamic.
  We also need to provide each component with a "key",
  which will be explained later.
-->
<todo-item
  v-for="item in groceryList"
  v-bind:todo="item"
  v-bind:key="item.id">
</todo-item>
  </ol>
</div>

    Vue.component('todo-item', {
  props: ['todo'],
  template: '<li>{{ todo.text }}</li>'
})
var app7 = new Vue({
  el: '#app-7',
  data: {
groceryList: [
  { id: 0, text: 'Vegetables' },
  { id: 1, text: 'Cheese' },
  { id: 2, text: 'Whatever else humans are supposed to eat' }
]
  }
})

  • Vue.component registers a component globally. Make a component available only in the scope of another instance/component by registering it with components instance option 

    var Child = {
  template: '<div>A custom component!</div>'
}
new Vue({
  // ...
  components: {
// <my-component> will only be available in parent's template
'my-component': Child
  }
})
  • all lowercase and must contain a hyphen

    • Although these can be used in string templates 

        components: {
'kebab-cased-component': { /* ... */ },
camelCasedComponent: { /* ... */ },
PascalCasedComponent: { /* ... */ }
  }

  • Component should be used in a way that has 3 parts

    props
    pass from parent to component
    events
    set parent based on events triggered from component
    slots
    customize, from parent, the look of the component
    <my-component
  :foo="baz"
  :bar="qux"
  @event-a="doThis"
  @event-b="doThat"
>
  <img slot="icon" src="...">
  <p slot="main-text">Hello!</p>
</my-component>
9.1.4.2. DOM Template Parsing Caveats

Becuase <ul>, <ol>, <table> and <select> need to have certain elements appearing inside them and some elements such as <option> can only appear inside certain other elements, it will lead to parsing issues if you do this 

<table>
  <my-row>...</my-row>
</table>

So better to do this 

<table>
  <tr is="my-row"></tr>
</table>

camelCase vs. kebab-case

These limitations do not apply if you are using string templates from one of the following sources:

  • <script type='text/x-template'>
  • JavaScript inline template strings
  • .vue components

So, prefer using string templates whenever possible.

9.1.4.3. Option data has to be a function

data is the initialization part that defines extra local data properties that are only available in the component.

3 simple-counter's have their own initializing part and scopes. 

<div id="example-2">
  <simple-counter></simple-counter>
  <simple-counter></simple-counter>
  <simple-counter></simple-counter>
</div>

Vue.component('simple-counter', {
  template: '<button v-on:click="counter += 1">{{ counter }}</button>',
  data: function () {
    return {
      counter: 0
    }
  }
})
new Vue({
  el: '#example-2'
})
9.1.4.4. Option filters
  • Filters provide text formatting
<!-- in mustaches -->
{{ message | capitalize }}
<!-- in v-bind -->
<div v-bind:id="rawId | formatId"></div>

<!-- can be chained -->
{{ message | filterA | filterB }}

<!-- can have arguments -->
{{ message | filterA('arg1', arg2) }}

filters: {
  capitalize: function (value) {
    if (!value) return ''
    value = value.toString()
    return value.charAt(0).toUpperCase() + value.slice(1)
  }
}

// define globally
Vue.filter('capitalize', function (value) {
  if (!value) return ''
  value = value.toString()
  return value.charAt(0).toUpperCase() + value.slice(1)
})
9.1.4.5. Props down

Pass down prop from parent to child using child component's prop option.

When the parent property updates, it will flow down to the child but not the other way around. 

Vue.component('child', {
  // declare the props
  props: ['message'],
  // like data, the prop can be used inside templates and
  // is also made available in the vm as this.message
  template: '<span>{{ message }}</span>'
})

// parent page
<child message="hello!"></child>

// dynamic binding a parent variable
<div>
  <input v-model="parentMsg">
  <br>
  <child v-bind:my-message="parentMsg"></child>
</div>

// pass all properties of the parent as props to the child
todo: {
  text: 'Learn Vue',
  isComplete: false
}

<todo-item v-bind="todo"></todo-item>

<todo-item
  v-bind:text="todo.text"
  v-bind:is-complete="todo.isComplete"
></todo-item>

Literal vs Dynamic 

<!-- this passes down a plain string "1" -->
<comp some-prop="1"></comp>

<!-- this passes down an actual number -->
<comp v-bind:some-prop="1"></comp>
  1. Don't mutate a prop of child component

    You shouldn't mutate a prop inside a child component. Refer to vue:data:mutation

    Because objects and arrays in JavaScript are passed by reference. So mutating the object or array inside the child will affect parent state.

    Instead you should do the following:

    • Define a local data prop that uses the prop's initial value as its initial value
    props: ['initialCounter'],
data: function () {
  return { counter: this.initialCounter }
}
    • Define a computed prop that is computed from the prop's value
    props: ['size'],
computed: {
  normalizedSize: function () {
    return this.size.trim().toLowerCase()
  }
}
  2. prop validation

    When prop validation fails, Vue will produce a console warning (if using the development build). Note that props are validated before a component instance is created, so within default or validator functions, instance properties such as from data, computed, or methods will not be available. 

    Vue.component('example', {
  props: {
    // basic type check (`null` means accept any type)
    propA: Number,

    // multiple possible types
    propB: [String, Number],

    // a required string
    propC: {
      type: String,
      required: true
    },

    // a number with default value
    propD: {
      type: Number,
      default: 100
    },

    // object/array defaults should be returned from a
    // factory function
    propE: {
      type: Object,
      default: function () {
        return { message: 'hello' }
      }
    },

    // custom validator function
    propF: {
      validator: function (value) {
        return value > 10
      }
    }
  }
})
    1. type

      type can be one of the following instructors String Number Boolean Function Object Array Symbol

      In addition, type can also be a custom constructor function and the assertion will be made with an instanceof check.

9.1.4.6. Events Up

vue:event:component

9.1.4.7. DOM template
  • In non-string templates, camelCased prop names need to use their kebab-case equivalents

  • Refer to see what vue:string template is 

    Vue.component('child', {
  // camelCase in JavaScript
  props: ['myMessage'],
  template: '<span>{{ myMessage }}</span>'
})

    <!-- kebab-case in HTML -->
<child my-message="hello!"></child>
9.1.4.8. Except class and style :: parent attributes overwrite child component

Child component bs-date-input has this template <input type="date" class="form-control">

But the child component is called from parent with the type attribute as well. 

<bs-date-input
  data-3d-date-picker="true"
  class="date-picker-theme-dark"
  type="large"
></bs-date-input>

The final result is type="large" because the value provided to the component will replace the value set by the component.

Except class and style attributes.

9.1.4.9. Slots

Parent content will be discarded unless the child component template contains at least one <slot> outlet.

  1. Basics

    my-component template: 

    <div>
  <h2>I'm the child title</h2>
  <slot>
    This will only be displayed if there is no content
    to be distributed.
  </slot>
</div>

    the parent 

    <div>
  <h1>I'm the parent title</h1>
  <my-component>
    <p>This is some original content</p>
    <p>This is some more original content</p>
  </my-component>
</div>

    Result 

    <div>
  <h1>I'm the parent title</h1>
  <div>
    <h2>I'm the child title</h2>
    <p>This is some original content</p>
    <p>This is some more original content</p>
  </div>
</div>
  2. Multiple slots

    app-layout component with template 

    <div class="container">
  <header>
    <slot name="header"></slot>
  </header>
  <main>
    <slot></slot>
  </main>
  <footer>
    <slot name="footer"></slot>
  </footer>
</div>

    Parent 

    <app-layout>
  <h1 slot="header">Here might be a page title</h1>
  <p>A paragraph for the main content.</p>
  <p>And another one.</p>
  <p slot="footer">Here's some contact info</p>
</app-layout>

    Result 

    <div class="container">
  <header>
    <h1>Here might be a page title</h1>
  </header>
  <main>
    <p>A paragraph for the main content.</p>
    <p>And another one.</p>
  </main>
  <footer>
    <p>Here's some contact info</p>
  </footer>
</div>
  3. Scoped slot

    From parent, create a temporary variable with an alias name which holds the props object passed from the child (props.text defined in child)

    This provides a way for parent to customize the look of a component.

    child component 

    <div class="child">
  <slot text="hello from child"></slot>
</div>

    Parent 

    <div class="parent">
  <child>
    <template slot-scope="props">
      <span>hello from parent</span>
      <span>{{ props.text }}</span>
    </template>
  </child>
</div>

    Result 

    <div class="parent">
  <div class="child">
    <span>hello from parent</span>
    <span>hello from child</span>
  </div>
</div>

    In 2.5.0+, slot-scope is no longer limited to <template> and can be used on any element or component.

    Number of slots in parent do not need to match number of slots in component.

    Parent :: customize the look for each list item 

    <my-awesome-list :items="items">
  <!-- scoped slot can be named too -->
  <li
    slot="item"
    slot-scope="props"
    class="my-fancy-item">
    {{ props.text }}
  </li>
</my-awesome-list>

    my-awesome-list component template 

    <ul>
  <slot name="item"
    v-for="item in items"
    :text="item.text">
    <!-- fallback content here -->
  </slot>
</ul>
  4. Destructuring for slot-scope
    <child>
  <span slot-scope="{ text }">{{ text }}</span>
</child>
9.1.4.10. Dynamic component, v-bind:is, <component>, <keep-alive>
var vm = new Vue({
  el: '#example',
  data: {
    currentView: 'home'
  },
  components: {
    home: { /* ... */ },
    posts: { /* ... */ },
    archive: { /* ... */ }
  }
})

<component v-bind:is="currentView">
  <!-- component changes when vm.currentView changes! -->
</component>

If <transition> is used, wrap <keep-alive> inside <transition> 

<keep-alive>
  <component :is="currentView">
    <!-- inactive components will be cached! -->
  </component>
</keep-alive>
9.1.4.11. Get Child Component One Time

$refs are only populated after the component has been rendered and it is not reactive. 

<div id="parent">
  <user-profile ref="profile"></user-profile>
</div>

var parent = new Vue({ el: '#parent' })
// access child component instance
var child = parent.$refs.profile
9.1.4.12. Async Components

Define component as a factory function that asynchronously resolves the component definition. The factory function will only be triggered when the component actually needs to be rendered and will cache the result for future re-renders.

Usually, component is defined using a name and an object of props.

A factory function can be used: 

Vue.component('async-example', function (resolve, reject) {
  setTimeout(function () {
    // Pass the component definition to the resolve callback
    resolve({
      template: '<div>I am async!</div>'
    })
  }, 1000)
})

Webpack example 

Vue.component('async-webpack-example', function (resolve) {
  // This special require syntax will instruct Webpack to
  // automatically split your built code into bundles which
  // are loaded over Ajax requests.
  require(['./my-async-component'], resolve)
})
  1. Component factory function can return a Promise

    (with Webpack) 

    Vue.component(
  'async-webpack-example',
  // The `import` function returns a `Promise`.
  () => import('./my-async-component')
)

new Vue({
  // ...
  components: {
    'my-component': () => import('./my-async-component')
  }
})
  2. Component factory function can also return an object
    const AsyncComp = () => ({
  // The component to load. Should be a Promise
  component: import('./MyComp.vue'),
  // A component to use while the async component is loading
  loading: LoadingComp,
  // A component to use if the load fails
  error: ErrorComp,
  // Delay before showing the loading component. Default: 200ms.
  delay: 200,
  // The error component will be displayed if a timeout is
  // provided and exceeded. Default: Infinity.
  timeout: 3000
})

    Refer to vue:router

9.1.4.13. Recursive and Circular Reference

A component can recursively invoke itself in its own template

When a component is registered globally using Vue.component, the global ID is auto set as the component's name option 

Vue.component('unique-name-of-my-component', {
  // ...
})

name: 'unique-name-of-my-component'
  1. Recursive
    name: 'stack-overflow',
template: '<div><stack-overflow></stack-overflow></div>
  2. Circular reference

    tree-folder component 

    <p>
  <span>{{ folder.name }}</span>
  <tree-folder-contents :children="folder.children"/>
</p>

    tree-folder-contents component 

    <ul>
  <li v-for="child in children">
    <tree-folder v-if="child.children" :folder="child"/>
    <span v-else>{{ child.name }}</span>
  </li>
</ul>

    tree-folder needs tree-folder-contents and tree-folder-contents needs tree-folder.

    It's ok if no module system is used to import components (Webpack or Browserify).

    You should do this 

    beforeCreate: function () {
  this.$options.components.TreeFolderContents = require('./tree-folder-contents.vue')
}
9.1.4.14. inline-template vue:inline-template

It makes the inner content as the component's template. Don't use it 

<my-component inline-template>
  <div>
    <p>These are compiled as the component's own template.</p>
    <p>Not parent's transclusion content.</p>
  </div>
</my-component>
9.1.4.15. X-Templates vue:x-template

Don't use 

<script type="text/x-template" id="hello-world-template">
  <p>Hello hello hello</p>
</script>

Vue.component('hello-world', {
  template: '#hello-world-template'
})

9.1.5. Template Syntax

9.1.5.1. String template vs DOM template

  • String template vue:string template 

    Vue.component('my-component', {
  template: '<p>This is a String template, it iss passed as a string to the component.</p>'
})

  • DOM template 

    <body>
  <div id="app"> <!-- your App is runnning in this div --->
<my-component></my-component>
  </div>

  <template id="template-for-my-component">
{{ message }}
  </template>
</body>
  • vue:x-template
9.1.5.2. v-once

one-time interpolations and remain cached 

<span v-once>This will never change: {{ msg }}</span>

Vue.component('terms-of-service', {
    template: '\
   <div v-once>\
     <h1>Terms of Service</h1>\
     ... a lot of static content ...\
   </div>\
     '
})
9.1.5.3. v-html Raw HTML

Can't be used for components. Watch out for XSS 

<div v-html="rawHtml"></div>
9.1.5.4. v-bind:attributeName, :attributeName 
<!-- bool is a bit different. If isButtonDisabled has the value of null, undefined or false, the disabled attribute will not be rendered -->
<button v-bind:disabled="isButtonDisabled">Button</button>
<button :disabled="isButtonDisabled">Button</button>

<!-- multiple attributes binding -->
<img v-bind="{
class: imgClass,
src: products[2].image,
alt: products[2].alt
}">
9.1.5.5. Expression
Some JavaScript expressions are allowed
search allowedGlobals in Vue Core
(no term)
Evaluated in the data scope of the owner Vue instance
(no term)
One binding one expression!
(no term)
Don't access user defined globals in template expressions
(no term)
Value of a directive attribute v-* usually is a single JavaScript expression
{{ number + 1 }}
{{ ok ? 'YES' : 'NO' }}
{{ message.split('').reverse().join('') }}
<div v-bind:id="'list-' + id"></div>
9.1.5.6. Directive Arguments v-bind:href, v-bind:class, v-on:click
  1. v-bind:href, v-on:click 
    <a v-bind:href="url"> ... </a>
// bind element's href attribute to the value of the expression url
<a v-on:click="doSomething"> ... </a>
// shorthand
<a @click="doSomething">...</a>
  2. v-bind:class
    <div class="static"
     v-bind:class="{ active: isActive, 'text-danger': hasError }">
</div>
<!-- <div class="static active"></div> -->

    data: {
    isActive: true,
    hasError: false
}

    Or 

    <div v-bind:class="classObject"></div>

    data: {
    classObject: {
    active: true,
    'text-danger': false
    }
}

// or even
data: {
    isActive: true,
    error: null
},
computed: {
    classObject: function () {
    return {
        active: this.isActive && !this.error,
        'text-danger': this.error && this.error.type === 'fatal'
    }
    }
}

    Array 

    <div :class="[activeClass, errorClass]"></div>
<!--
data: {
  activeClass: 'active',
  errorClass: 'text-danger'
}
-->

<div v-bind:class="[isActive ? activeClass : '', errorClass]"></div>

<div v-bind:class="[{ active: isActive }, errorClass]"></div>

    When you use the class attribute on a custom component, those classes will be added to the component’s root element. Existing classes on this element will not be overwritten. 

    Vue.component('my-component', {
  template: '<p class="foo bar">Hi</p>'
})

// <my-component class="baz boo"></my-component>

// the rendered HTML will be:
// <p class="foo bar baz boo">Hi</p>
// the same is also true for class bindings:

// <my-component v-bind:class="{ active: isActive }"></my-component>

// Rendered:
// <p class="foo bar active">Hi</p>
  3. v-bind:style
    • CSS vendor-prefixes are auto detected and added when v-bind:style is used
    • Use either camelCase with hypen and quotes or kebab-case with hyphen and quotes for CSS property name
    <div v-bind:style="{ color: activeColor, fontSize: fontSize + 'px' }"></div>

<!-- kebob case -->
<div v-bind:style="{ color: activeColor, 'font-size': fontSize + 'px' }"></div>

<!--
data: {
  activeColor: 'red',
  fontSize: 30
}
-->

<div v-bind:style="styleObject"></div>
<!--
data: {
  styleObject: {
    color: 'red',
    fontSize: '13px'
  }
}
-->

    Array 

    <div v-bind:style="[baseStyles, overridingStyles]"></div>
9.1.5.7. Dynamic Directive Argument
myAttributeNameVar
expression should return string or null. Any other non-string value triggers a warning
null
remove the binding
Don't use spaces or quotes in the expression
DON'T <a v-bind:['foo' + bar]="value"> ... </a>
In-DOM templates (templates in an HTML file)
expression will be converted to lowercase
<a v-bind:[myAttributeNameVar]="url"> ... </a>
<a @[myeventnamevar]="doSomething">...</a>
9.1.5.8. Modifiers v-on:submit.prevent
  • Refer to vue:event:modifiers

  • Example Calls event.preventDefault() on the triggered event 

    <form v-on:submit.prevent="onSubmit"> ... </form>
9.1.5.9. v-if, key attribute, v-show

  • v-if, v-else, v-else-if 

    <h1 v-if="ok">Yes</h1>
<h1 v-else>No</h1>

    • v-else must immediately follow a v-if or v-else-if element 

        <div v-if="Math.random() > 0.5">
Now you see me
  </div>
  <div v-else>
Now you don't
  </div>

    • v-else-if since 2.1.0+ 

        <div v-if="type === 'A'">
A
  </div>
  <div v-else-if="type === 'B'">
B
  </div>
  <div v-else-if="type === 'C'">
C
  </div>
  <div v-else>
Not A/B/C
  </div>
  • When used together with v-if, v-for has a higher priority than v-if.

  • Group with v-if on <template>. template element will not be included 

    <template v-if="ok">
  <h1>Title</h1>
  <p>Paragraph 1</p>
  <p>Paragraph 2</p>
</template>

  • key attribute marks an element unique which means elements shouldn't be reused unless it has the same key 

    <template v-if="loginType === 'username'">
  <label>Username</label>
  <input placeholder="Enter your username" key="username-input">
</template>
<template v-else>
  <label>Email</label>
  <input placeholder="Enter your email address" key="email-input">
</template>
  • v-show toggles the element’s display CSS property based on the truthy-ness of the expression value
    • v-show doesn’t support the <template> element, nor does it work with v-else.
    • v-if vs v-show
      • Expensive to toggle v-if, so use it when the condition is unlikely to change over time
      • v-if is lazy means if it's false on initial render, it will not do anything, the conditional block won't be rendered until the condition becomes true for the first time
      • v-show always render
      • Use v-show if it needs to be toggled very often
    • <h1 v-show="ok">Hello!</h1>
9.1.5.10. v-for
  1. Basics

    v-for has full access to parent scope properties 

    // array
<ul id="example-1">
  <li v-for="item in items">
    {{ item.message }}
  </li>
</ul>
var example1 = new Vue({
  el: '#example-1',
  data: {
    items: [
      { message: 'Foo' },
      { message: 'Bar' }
    ]
  }
})

<ul id="example-2">
  <li v-for="(item, index) in items">
    {{ parentMessage }} - {{ index }} - {{ item.message }}
  </li>
</ul>

var example2 = new Vue({
  el: '#example-2',
  data: {
    parentMessage: 'Parent',
    items: [
      { message: 'Foo' },
      { message: 'Bar' }
    ]
  }
})

<div v-for="item of items"></div>

// object
<ul id="v-for-object" class="demo">
  <li v-for="value in object">
    {{ value }}
  </li>
</ul>

new Vue({
  el: '#v-for-object',
  data: {
    object: {
      firstName: 'John',
      lastName: 'Doe',
      age: 30
    }
  }
})

<div v-for="(value, key) in object">
  {{ key }}: {{ value }}
</div>

<div v-for="(value, key, index) in object">
  {{ index }}. {{ key }}: {{ value }}
</div>

    The default mode is only suitable when your list render output does not reply on child component state or temporary DOM state (e.g. form input values)

    Provide a unqiue key attribute for each item. 

    <div v-for="item in items" :key="item.id">
  <!-- content -->
</div>

    Refer to vue:data for mutation and non-mutating methods

  2. Filtered/Sorted
    <li v-for="n in evenNumbers">{{ n }}</li>

data: {
  numbers: [ 1, 2, 3, 4, 5 ]
},
computed: {
  evenNumbers: function () {
    return this.numbers.filter(function (number) {
      return number % 2 === 0
    })
  }
}

    In case when computed properties are not feasible e.g. inside nested v-for loops 

    <li v-for="n in even(numbers)">{{ n }}</li>
data: {
  numbers: [ 1, 2, 3, 4, 5 ]
},
methods: {
  even: function (numbers) {
    return numbers.filter(function (number) {
      return number % 2 === 0
    })
  }
}
  3. range
    <div>
  <span v-for="n in 10">{{ n }} </span>
</div>
  4. v-for on <template>
    <ul>
  <template v-for="item in items">
    <li>{{ item.msg }}</li>
    <li class="divider"></li>
  </template>
</ul>
  5. v-for with v-if
    <li v-for="todo in todos" v-if="!todo.isComplete">
  {{ todo }}
</li>

// don't execute v-for at all
<ul v-if="todos.length">
  <li v-for="todo in todos">
    {{ todo }}
  </li>
</ul>
<p v-else>No todos left!</p>
  6. v-for with a component
    <my-component v-for="item in items" :key="item.id"></my-component>

// pass data to component
<my-component
  v-for="(item, index) in items"
  v-bind:item="item"
  v-bind:index="index"
  v-bind:key="item.id"
></my-component>
  7. Example

    Because it has to be at least one <li> in <ul> to be valid. is="todo-item" is needed. 

    <div id="todo-list-example">
  <input
    v-model="newTodoText"
    v-on:keyup.enter="addNewTodo"
    placeholder="Add a todo"
  >
  <ul>
    <li
      is="todo-item"
      v-for="(todo, index) in todos"
      v-bind:key="todo.id"
      v-bind:title="todo.title"
      v-on:remove="todos.splice(index, 1)"
    ></li>
  </ul>
</div>

Vue.component('todo-item', {
  template: '\
    <li>\
      {{ title }}\
      <button v-on:click="$emit(\'remove\')">X</button>\
    </li>\
  ',
  props: ['title']
})
new Vue({
  el: '#todo-list-example',
  data: {
    newTodoText: '',
    todos: [
      {
        id: 1,
        title: 'Do the dishes',
      },
      {
        id: 2,
        title: 'Take out the trash',
      },
      {
        id: 3,
        title: 'Mow the lawn'
      }
    ],
    nextTodoId: 4
  },
  methods: {
    addNewTodo: function () {
      this.todos.push({
        id: this.nextTodoId++,
        title: this.newTodoText
      })
      this.newTodoText = ''
    }
  }
})
9.1.5.11. Event, v-on, @attributeName
  1. Basics
    <div id="example-1">
  <button v-on:click="counter += 1">Add 1</button>
  <p>The button above has been clicked {{ counter }} times.</p>
</div>
var example1 = new Vue({
  el: '#example-1',
  data: {
    counter: 0
  }
})
  2. Method calling
    <div id="example-2">
  <!-- `greet` is the name of a method defined below -->
  <button v-on:click="greet">Greet</button>
</div>
var example2 = new Vue({
  el: '#example-2',
  data: {
    name: 'Vue.js'
  },
  // define methods under the `methods` object
  methods: {
    greet: function (event) {
      // `this` inside methods points to the Vue instance
      alert('Hello ' + this.name + '!')
      // `event` is the native DOM event
      if (event) {
        alert(event.target.tagName)
      }
    }
  }
})
// you can invoke methods in JavaScript too
example2.greet() // => 'Hello Vue.js!'
  3. Inline with $event
    <div id="example-3">
  <button v-on:click="say('hi')">Say hi</button>
  <button v-on:click="say('what')">Say what</button>
</div>
new Vue({
  el: '#example-3',
  methods: {
    say: function (message) {
      alert(message)
    }
  }
})

<button v-on:click="warn('Form cannot be submitted yet.', $event)">
  Submit
</button>
methods: {
  warn: function (message, event) {
    // now we have access to the native event
    if (event) event.preventDefault()
    alert(message)
  }
}
  4. Event Modifiers vue:event:modifiers
    <!-- the click event's propagation will be stopped -->
<a v-on:click.stop="doThis"></a>

<!-- the submit event will no longer reload the page -->
<form v-on:submit.prevent="onSubmit"></form>

<!-- modifiers can be chained :: chain stop and prevent. -->
<a v-on:click.stop.prevent="doThat"></a>

<!-- chain order matters. @click.prevent.self will prevent all clicks while @click.self.prevent will only prevent clicks on the element itself -->

<!-- just the modifier -->
<form v-on:submit.prevent></form>

<!-- use capture mode when adding the event listener -->
<!-- i.e. an event targeting an inner element is handled here before being handled by that element -->
<div v-on:click.capture="doThis">...</div>

<!-- only trigger handler if event.target is the element itself -->
<!-- i.e. not from a child element -->
<div v-on:click.self="doThat">...</div>

<!-- the click event will be triggered at most once. 2.1.4+ -->
<a v-on:click.once="doThis"></a>
  5. Event Key Modifiers vue:event:key modifiers
    <!-- only call vm.submit() when the keyCode is 13 -->
<input v-on:keyup.13="submit">

<!-- also works for shorthand -->
<input @keyup.enter="submit">

    Full list :: .enter, .tab, .delete (both Delete and Backspace), .esc, .space, .up, .down, .left, .right

    Define custom key modifier aliase 

    // enable v-on:keyup.f1
Vue.config.keyCodes.f1 = 112

    Directly use any valid key names provided by KeyboardEvent.key as modifiers by converting them to kebab-case. e.g. the handler will only be called if $event.key = 'PageDown'. 

    <input @keyup.page-down="onPageDown">

    .ctrl, .alt, .shift, .meta (windows key or Mac's command key) 

    <!-- Alt + C -->
<input @keyup.alt.67="clear">

<!-- Ctrl + Click -->
<div @click.ctrl="doSomething">Do something</div>

    .exact 

    <!-- this will fire even if Alt or Shift is also pressed -->
<button @click.ctrl="onClick">A</button>

<!-- this will only fire when only Ctrl is pressed -->
<button @click.ctrl.exact="onCtrlClick">A</button>

    .left, .right, .middle

  6. Event from child component. .sync vue:event:component
    <div id="counter-event-example">
  <p>{{ total }}</p>
  <button-counter v-on:increment="incrementTotal"></button-counter>
  <button-counter v-on:increment="incrementTotal"></button-counter>
</div>

Vue.component('button-counter', {
  template: '<button v-on:click="incrementCounter">{{ counter }}</button>',
  data: function () {
    return {
      counter: 0
    }
  },
  methods: {
    incrementCounter: function () {
      this.counter += 1
      this.$emit('increment')
    }
  },
})
new Vue({
  el: '#counter-event-example',
  data: {
    total: 0
  },
  methods: {
    incrementTotal: function () {
      this.total += 1
    }
  }
})

    vm.$on can only listen events on teh current vm. But in order to listen from child components, you will need v-on in parent template 

    vm.$on('test', function (msg) {
  console.log(msg)
})
vm.$emit('test', 'hi')
// => "hi"

    In parent, when adding a native even (e.g. click) on a component, @click.native="parentMethod" is required.

    .sync By default one-way data flow from parent to child. There's time when modifying parent prop is desired. But try to avoid this. 

    <comp :foo.sync="bar"></comp>

// got expanded to

<comp :foo="bar" @update:foo="val => bar = val"></comp>

// For the child component to update foo‘s value, it needs to explicitly emit an event instead of mutating the prop:

this.$emit('update:foo', newValue)
  7. Event in Form, v-model vue:event:v-model
    <input v-model="something">
// is equivalent of
<input
  v-bind:value="something"
  v-on:input="something = $event.target.value">

// For a component
<custom-input
  :value="something"
  @input="value => { something = value }">
</custom-input>

    Custom form input The key point is the component has to have props: ['value'] and emit an input event with the new value

    1. text
      <script src="https://cdn.rawgit.com/chrisvfritz/5f0a639590d6e648933416f90ba7ae4e/raw/974aa47f8f9c5361c5233bd56be37db8ed765a09/currency-validator.js"></script>
<div id="app">
  <currency-input 
    label="Price" 
    v-model="price"
  ></currency-input>
  <currency-input 
    label="Shipping" 
    v-model="shipping"
  ></currency-input>
  <currency-input 
    label="Handling" 
    v-model="handling"
  ></currency-input>
  <currency-input 
    label="Discount" 
    v-model="discount"
  ></currency-input>

  <p>Total: ${{ total }}</p>
</div>

Vue.component('currency-input', {
  template: '\
    <div>\
      <label v-if="label">{{ label }}</label>\
      $\
      <input\
        ref="input"\
        v-bind:value="value"\
        v-on:input="updateValue($event.target.value)"\
        v-on:focus="selectAll"\
        v-on:blur="formatValue"\
      >\
    </div>\
  ',
  props: {
    value: {
      type: Number,
      default: 0
    },
    label: {
      type: String,
      default: ''
    }
  },
  mounted: function () {
    this.formatValue()
  },
  methods: {
    updateValue: function (value) {
      var result = currencyValidator.parse(value, this.value)
      if (result.warning) {
        this.$refs.input.value = result.value
      }
      this.$emit('input', result.value)
    },
    formatValue: function () {
      this.$refs.input.value = currencyValidator.format(this.value)
    },
    selectAll: function (event) {
      // Workaround for Safari bug
      // http://stackoverflow.com/questions/1269722/selecting-text-on-focus-using-jquery-not-working-in-safari-and-chrome
      setTimeout(function () {
        event.target.select()
      }, 0)
    }
  }
})

new Vue({
  el: '#app',
  data: {
    price: 0,
    shipping: 0,
    handling: 0,
    discount: 0
  },
  computed: {
    total: function () {
      return ((
        this.price * 100 + 
        this.shipping * 100 + 
        this.handling * 100 - 
        this.discount * 100
      ) / 100).toFixed(2)
    }
  }
})
    2. checkbox

      By default, v-model on a component uses value as the prop and input as the event, but some input types such as checkboxes and radio buttons may want to use the value prop for a different purpose. Using the model option can avoid the conflict in such cases 

      Vue.component('my-checkbox', {
  model: {
    prop: 'checked',
    event: 'change'
  },
  props: {
    checked: Boolean,
    // this allows using the `value` prop for a different purpose
    value: String
  },
  // ...
})

<my-checkbox v-model="foo" value="some value"></my-checkbox>

// is equivalent to
<my-checkbox
  :checked="foo"
  @change="val => { foo = val }"
  value="some value">
</my-checkbox>
  8. Use Event to transfer data between components that are not parent-child
    var bus = new Vue()
bus.$emit('id-selected', 1)
// in component B's created hook
bus.$on('id-selected', function (id) {
  // ...
})

    If it's more complex, refer to vue:state management

9.1.5.12. Form input, v-model

v-model will ignore the initial value, checked or selected.

  1. textarea
    <span>Multiline message is:</span>
<p style="white-space: pre-line;">{{ message }}</p>  <br>
<textarea v-model="message" placeholder="add multiple lines"></textarea>
  2. checkbox
    <input type="checkbox" id="checkbox" v-model="checked">
<label for="checkbox">{{ checked }}</label>

<!-- `toggle` is either true or false -->
<input type="checkbox" v-model="toggle">

// multiple checkboxes

<div id='example-3'>
  <input type="checkbox" id="jack" value="Jack" v-model="checkedNames">
  <label for="jack">Jack</label>
  <input type="checkbox" id="john" value="John" v-model="checkedNames">
  <label for="john">John</label>
  <input type="checkbox" id="mike" value="Mike" v-model="checkedNames">
  <label for="mike">Mike</label>
  <br>
  <span>Checked names: {{ checkedNames }}</span>
</div>

new Vue({
  el: '#example-3',
  data: {
    checkedNames: []
  }
})
  3. radio
    <input type="radio" id="one" value="One" v-model="picked">
<label for="one">One</label>
<br>
<input type="radio" id="two" value="Two" v-model="picked">
<label for="two">Two</label>
<br>
<span>Picked: {{ picked }}</span>

<!-- `picked` is a string "a" when checked -->
<input type="radio" v-model="picked" value="a">
  4. select
    <select v-model="selected">
  <option disabled value="">Please select one</option>
  <option>A</option>
  <option>B</option>
  <option>C</option>
</select>
<span>Selected: {{ selected }}</span>

new Vue({
  el: '...',
  data: {
    selected: ''
  }
})

<!-- `selected` is a string "abc" when selected -->
<select v-model="selected">
  <option value="abc">ABC</option>
</select>

// select multiple
<select v-model="selected" multiple>
  <option>A</option>
  <option>B</option>
  <option>C</option>
</select>
<br>
<span>Selected: {{ selected }}</span>

// select v-for
<select v-model="selected">
  <option v-for="option in options" v-bind:value="option.value">
    {{ option.text }}
  </option>
</select>
<span>Selected: {{ selected }}</span>

new Vue({
  el: '...',
  data: {
    selected: 'A',
    options: [
      { text: 'One', value: 'A' },
      { text: 'Two', value: 'B' },
      { text: 'Three', value: 'C' }
    ]
  }
})
  5. Use v-bind with v-model to prepopulate
    <input
  type="checkbox"
  v-model="toggle"
  v-bind:true-value="a"
  v-bind:false-value="b"
>

// when checked:
vm.toggle === vm.a
// when unchecked:
vm.toggle === vm.b

<input type="radio" v-model="pick" v-bind:value="a">
// when checked:
vm.pick === vm.a

<select v-model="selected">
  <!-- inline object literal -->
  <option v-bind:value="{ number: 123 }">123</option>
</select>
  6. v-model modifiers
    .lazy
<!-- synced after "change" instead of "input" -->
<input v-model.lazy="msg" >

.number :: typecast as a number
<input v-model.number="age" type="number">

.trim
<input v-model.trim="msg">
  7. Event in v-model

    vue:event:v-model

9.1.6. Render Functions & JSX

9.1.6.1. Render function

To avoid duplicating <slot> in string template 

<anchored-heading :level="1">Hello world!</anchored-heading>

<script type="text/x-template" id="anchored-heading-template">
  <h1 v-if="level === 1">
    <slot></slot>
  </h1>
  <h2 v-else-if="level === 2">
    <slot></slot>
  </h2>
  <h3 v-else-if="level === 3">
    <slot></slot>
  </h3>
  <h4 v-else-if="level === 4">
    <slot></slot>
  </h4>
  <h5 v-else-if="level === 5">
    <slot></slot>
  </h5>
  <h6 v-else-if="level === 6">
    <slot></slot>
  </h6>
</script>

<!--
Vue.component('anchored-heading', {
  template: '#anchored-heading-template',
  props: {
    level: {
  type: Number,
  required: true
    }
  }
})
-->

Use render function (reactive) 

Vue.component('anchored-heading', {
  render: function (createElement) {
    return createElement(
  'h' + this.level,   // tag name
  this.$slots.default // array of children
    )
  },
  props: {
    level: {
  type: Number,
  required: true
    }
  }
})
  1. createElement Arguments
    // @returns {VNode}
createElement(
  // {String | Object | Function}
  // An HTML tag name, component options, or function
  // returning one of these. Required.
  'div',

  // {Object}
  // A data object corresponding to the attributes
  // you would use in a template. Optional.
  {
    // (see details in the next section below)
  },

  // {String | Array}
  // Children VNodes, built using `createElement()`,
  // or using strings to get 'text VNodes'. Optional.
  [
    'Some text comes first.',
    createElement('h1', 'A headline'),
    createElement(MyComponent, {
      props: {
        someProp: 'foobar'
      }
    })
  ]
)

    {
  // Same API as `v-bind:class`
  'class': {
    foo: true,
    bar: false
  },
  // Same API as `v-bind:style`
  style: {
    color: 'red',
    fontSize: '14px'
  },
  // Normal HTML attributes
  attrs: {
    id: 'foo'
  },
  // Component props
  props: {
    myProp: 'bar'
  },
  // DOM properties
  domProps: {
    innerHTML: 'baz'
  },
  // Event handlers are nested under `on`, though
  // modifiers such as in `v-on:keyup.enter` are not
  // supported. You'll have to manually check the
  // keyCode in the handler instead.
  on: {
    click: this.clickHandler
  },
  // For components only. Allows you to listen to
  // native events, rather than events emitted from
  // the component using `vm.$emit`.
  nativeOn: {
    click: this.nativeClickHandler
  },
  // Custom directives. Note that the binding's
  // oldValue cannot be set, as Vue keeps track
  // of it for you.
  directives: [
    {
      name: 'my-custom-directive',
      value: '2',
      expression: '1 + 1',
      arg: 'foo',
      modifiers: {
        bar: true
      }
    }
  ],
  // Scoped slots in the form of
  // { name: props => VNode | Array<VNode> }
  scopedSlots: {
    default: props => createElement('span', props.text)
  },
  // The name of the slot, if this component is the
  // child of another component
  slot: 'name-of-slot',
  // Other special top-level properties
  key: 'myKey',
  ref: 'myRef'
}
  2. createElement Constraints

    All VNodes in the component tree must be unique. This is invalid 

    render: function (createElement) {
  var myParagraphVNode = createElement('p', 'hi')
  return createElement('div', [
    // Yikes - duplicate VNodes!
    myParagraphVNode, myParagraphVNode
  ])
}

    Use factory function 

    render: function (createElement) {
  return createElement('div',
  Array.apply(null, { length: 20 }).map(function () {
      return createElement('p', 'hi')
    })
  )
}
  3. render for v-if and v-for
  4. render for v-model
  5. redner for event & key modifiers
  6. render for <slot>

    Static slot contents as Arrays of VNodes from this.$slots 

    render: function (createElement) {
  // `<div><slot></slot></div>`
  return createElement('div', this.$slots.default)
}

    Scoped slot 

    render: function (createElement) {
  // `<div><slot :text="msg"></slot></div>`
  return createElement('div', [
    this.$scopedSlots.default({
      text: this.msg
    })
  ])
}

// pass scoped slots to a child component
render (createElement) {
  return createElement('div', [
    createElement('child', {
      // pass `scopedSlots` in the data object
      // in the form of { name: props => VNode | Array<VNode> }
      scopedSlots: {
        default: function (props) {
          return createElement('span', props.text)
        }
      }
    })
  ])
}
9.1.6.2. JSX

9.1.7. vm.computed property, vm.watch property

9.1.7.1. Computed property

  • Example 

    <div id="example">
  <p>Original message: "{{ message }}"</p>
  <p>Computed reversed message: "{{ reversedMessage }}"</p>
  <!-- same as 
  <p>Reversed message: "{{ reverseMessage()  }}"</p>
  -->
</div>

    var vm = new Vue({
  el: '#example',
  data: {
message: 'Hello'
  },
  computed: {
// a computed getter
reversedMessage: function () {
  // `this` points to the vm instance
  return this.message.split('').reverse().join('')
}
  }
  /* same as method
  methods: {
reverseMessage: function () {
  return this.message.split('').reverse().join('')
}
  '}
  */
})
  • vm.computed.reversedMessage is reactive to vm.$data.message and it's cached when there is no change

  • While method will run every time the view updates (re-render). Hence, never define computed property without refering to vm.$data e.g. Don't do this 

    computed: {
  now: function () {
return Date.now()
  }
}

  • Computed Setter and Getter 

    computed: {
  fullName: {
// getter
get: function () {
  return this.firstName + ' ' + this.lastName
},
// setter
set: function (newValue) {
  var names = newValue.split(' ')
  this.firstName = names[0]
  this.lastName = names[names.length - 1]
}
  }
}
9.1.7.2. vm.watch property - Watchers
  • vm.watch is to watch a property vm.data.propA change

  • Try to avoid using watch too much, use computed property instead Instead of 

    var vm = new Vue({
  el: '#demo',
  data: {
firstName: 'Foo',
lastName: 'Bar',
fullName: 'Foo Bar'
  },
  watch: {
firstName: function (newval, oldval) {
  this.fullName = newval + ' ' + this.lastName
},
lastName: function (newval, oldval) {
  this.fullName = this.firstName + ' ' + newval
}
  }
})

    Use 

    var vm = new Vue({
  el: '#demo',
  data: {
firstName: 'Foo',
lastName: 'Bar'
  },
  computed: {
fullName: function () {
  return this.firstName + ' ' + this.lastName
}
  }
})

  • Another example 

    <div id="watch-example">
  <p>
Ask a yes/no question:
<input v-model="question">
  </p>
  <p>{{ answer }}</p>
</div>

<!-- Since there is already a rich ecosystem of ajax libraries    -->
<!-- and collections of general-purpose utility methods, Vue core -->
<!-- is able to remain small by not reinventing them. This also   -->
<!-- gives you the freedom to use what you're familiar with. -->
<script src="https://cdn.jsdelivr.net/npm/axios@0.12.0/dist/axios.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/lodash@4.13.1/lodash.min.js"></script>

    var watchExampleVM = new Vue({
  el: '#watch-example',
  data: {
question: '',
answer: 'I cannot give you an answer until you ask a question!'
  },
  watch: {
// whenever question changes, this function will run
question: function (newQuestion) {
  this.answer = 'Waiting for you to stop typing...'
  this.getAnswer()
}
  },
  methods: {
// _.debounce is a function provided by lodash to limit how
// often a particularly expensive operation can be run.
// In this case, we want to limit how often we access
// yesno.wtf/api, waiting until the user has completely
// finished typing before making the ajax request. To learn
// more about the _.debounce function (and its cousin
// _.throttle), visit: https://lodash.com/docs#debounce
getAnswer: _.debounce(
  function () {
    if (this.question.indexOf('?') === -1) {
      this.answer = 'Questions usually contain a question mark. ;-)'
      return
    }
    this.answer = 'Thinking...'
    var vm = this
    axios.get('https://yesno.wtf/api')
      .then(function (response) {
    vm.answer = _.capitalize(response.data.answer)
      })
      .catch(function (error) {
    vm.answer = 'Error! Could not reach the API. ' + error
      })
  },
  // This is the number of milliseconds we wait for the
  // user to stop typing.
  500
)
  }
})

9.1.8. Transition & Animation

9.1.8.1. <transition> wrapper component
  • Wrap v-if, v-show, dynamic components and component root nodes
  1. CSS Transition
    <div id="demo">
  <button v-on:click="show = !show">
    Toggle
  </button>
  <transition name="fade">
    <p v-if="show">hello</p>
  </transition>
</div>

    new Vue({
  el: '#demo',
  data: {
    show: true
  }
})

    .fade-enter-active, .fade-leave-active {
  transition: opacity .5s
}
.fade-enter, .fade-leave-to /* .fade-leave-active below version 2.1.8 */ {
  opacity: 0
}

/* more examples using pure css
 * Enter and leave animations can use different
 * durations and timing functions.              
 */
.slide-fade-enter-active {
  transition: all .3s ease;
}
.slide-fade-leave-active {
  transition: all .8s cubic-bezier(1.0, 0.5, 0.8, 1.0);
}
.slide-fade-enter, .slide-fade-leave-to
/* .slide-fade-leave-active below version 2.1.8 */ {
  transform: translateX(10px);
  opacity: 0;
}
    • Classes
      *-enter
      before element inserted > one frame after element is inserted
      *-enter-active
      before element is inserted > removed when transition/animation finishes. Use it to define during, delay and easing curve for the entering transition.
      (no term)
      *-enter-to : one frame after element is inserted > transition/animation finishes.
      *-leave
      leave transition is triggered > after one frame
      *-leave-active
      leave transition is triggered > transition/animation finishes. Use it to define duration, delay and easing curve for the leaving transition.
      *-leave-to
      one frame after a leave transition is triggered > transition/animation finishes.
      (no term)
      *-enter is enter start, *-enter-to is enter end
      (no term)
      *-leave is leave start, *-leave-to is leave end
      (no term)
      v- prefix is used when <transition> has no name
  2. CSS Animations
    <div id="example-2">
  <button @click="show = !show">Toggle show</button>
  <transition name="bounce">
    <p v-if="show">Look at me!</p>
  </transition>
</div>

new Vue({
  el: '#example-2',
  data: {
    show: true
  }
})

.bounce-enter-active {
  animation: bounce-in .5s;
}
.bounce-leave-active {
  animation: bounce-in .5s reverse;
}
@keyframes bounce-in {
  0% {
    transform: scale(0);
  }
  50% {
    transform: scale(1.5);
  }
  100% {
    transform: scale(1);
  }
}
  3. Custom Transition Class

    Override the default class names such as Animate.css 

    <link href="https://cdn.jsdelivr.net/npm/animate.css@3.5.1" rel="stylesheet" type="text/css">
<div id="example-3">
  <button @click="show = !show">
    Toggle render
  </button>
  <transition
    name="custom-classes-transition"
    enter-active-class="animated tada"
    leave-active-class="animated bounceOutRight"
  >
    <p v-if="show">hello</p>
  </transition>
</div>
  4. prop
    <transition :duration="1000">...</transition>
<transition :duration="{ enter: 500, leave: 800 }">...</transition>
  5. event hooks

    When using JavaScript-only transitions, the done callbacks are required for the enter and leave hooks.

    1. events
      <transition
  v-on:before-enter="beforeEnter"
  v-on:enter="enter"
  v-on:after-enter="afterEnter"
  v-on:enter-cancelled="enterCancelled"
  v-on:before-leave="beforeLeave"
  v-on:leave="leave"
  v-on:after-leave="afterLeave"
  v-on:leave-cancelled="leaveCancelled"
>
  <!-- ... -->
</transition>
    2. methods
      // ...
methods: {
  // --------
  // ENTERING
  // --------
  beforeEnter: function (el) {
    // ...
  },
  // the done callback is optional when
  // used in combination with CSS
  enter: function (el, done) {
    // ...
    done()
  },
  afterEnter: function (el) {
    // ...
  },
  enterCancelled: function (el) {
    // ...
  },
  // --------
  // LEAVING
  // --------
  beforeLeave: function (el) {
    // ...
  },
  // the done callback is optional when
  // used in combination with CSS
  leave: function (el, done) {
    // ...
    done()
  },
  afterLeave: function (el) {
    // ...
  },
  // leaveCancelled only available with v-show
  leaveCancelled: function (el) {
    // ...
  }
}

      When javascript-only transistions, better to add v-bind:css="false"

    3. Example using Velocity.js
      <!--
Velocity works very much like jQuery.animate and is
a great option for JavaScript animations
-->
<script src="https://cdnjs.cloudflare.com/ajax/libs/velocity/1.2.3/velocity.min.js"></script>
<div id="example-4">
  <button @click="show = !show">
    Toggle
  </button>
  <transition
    v-on:before-enter="beforeEnter"
    v-on:enter="enter"
    v-on:leave="leave"
    v-bind:css="false"
  >
    <p v-if="show">
      Demo
    </p>
  </transition>
</div>

new Vue({
  el: '#example-4',
  data: {
    show: false
  },
  methods: {
    beforeEnter: function (el) {
      el.style.opacity = 0
    },
    enter: function (el, done) {
      Velocity(el, { opacity: 1, fontSize: '1.4em' }, { duration: 300 })
      Velocity(el, { fontSize: '1em' }, { complete: done })
    },
    leave: function (el, done) {
      Velocity(el, { translateX: '15px', rotateZ: '50deg' }, { duration: 600 })
      Velocity(el, { rotateZ: '100deg' }, { loop: 2 })
      Velocity(el, {
        rotateZ: '45deg',
        translateY: '30px',
        translateX: '30px',
        opacity: 0
      }, { complete: done })
    }
  }
})
  6. Transition on Initial Render

    Use appear attribute 

    // default only uses enter and leave
<transition appear>
  <!-- ... -->
</transition>

// specify more or custom CSS classes
<transition
  appear
  appear-class="custom-appear-class"
  appear-to-class="custom-appear-to-class" (2.1.8+)
  appear-active-class="custom-appear-active-class"
>
  <!-- ... -->
</transition>

// and custom javascript hooks
<transition
  appear
  v-on:before-appear="customBeforeAppearHook"
  v-on:appear="customAppearHook"
  v-on:after-appear="customAfterAppearHook"
  v-on:appear-cancelled="customAppearCancelledHook"
>
  <!-- ... -->
</transition>
  7. Transition between elements
    <transition>
  <button v-if="docState === 'saved'" key="saved">
    Edit
  </button>
  <button v-if="docState === 'edited'" key="edited">
    Save
  </button>
  <button v-if="docState === 'editing'" key="editing">
    Cancel
  </button>
</transition>

// It's equivalent to
<transition>
  <button v-bind:key="docState">
    {{ buttonMessage }}
  </button>
</transition>

// ...
computed: {
  buttonMessage: function () {
    switch (this.docState) {
      case 'saved': return 'Edit'
      case 'edited': return 'Save'
      case 'editing': return 'Cancel'
    }
  }
}
  8. Transition modes

    Default behavior is entering and leaving happens simultaneously. 

    <transition name="fade" mode="out-in">
  <!-- ... the buttons ... -->
</transition>

    in-out :: new element transitions in first then when complete, the current element transitions out. out-in :: current element transitions out first then when complete, the new element transitions in.

  9. Transition between dynamic components

    key attribute is not required 

    <transition name="component-fade" mode="out-in">
  <component v-bind:is="view"></component>
</transition>

new Vue({
  el: '#transition-components-demo',
  data: {
    view: 'v-a'
  },
  components: {
    'v-a': {
      template: '<div>Component A</div>'
    },
    'v-b': {
      template: '<div>Component B</div>'
    }
  }
})
9.1.8.2. Loop items transition transition-group
  • Attributes
    tag="span"
    Default is to animate each span child item inside the <transition-group>, this example change it to animate p elements
    key
    required
  1. Animate entering and leaving
    <div id="list-demo">
  <button v-on:click="add">Add</button>
  <button v-on:click="remove">Remove</button>
  <transition-group name="list" tag="p">
    <span v-for="item in items" v-bind:key="item" class="list-item">
  {{ item }}
    </span>
  </transition-group>
</div>

    new Vue({
  el: '#list-demo',
  data: {
    items: [1,2,3,4,5,6,7,8,9],
    nextNum: 10
  },
  methods: {
    randomIndex: function () {
  return Math.floor(Math.random() * this.items.length)
    },
    add: function () {
  this.items.splice(this.randomIndex(), 0, this.nextNum++)
    },
    remove: function () {
  this.items.splice(this.randomIndex(), 1)
    },
  }
})

    .list-item {
  display: inline-block;
  margin-right: 10px;
}
.list-enter-active, .list-leave-active {
  transition: all 1s;
}
.list-enter, .list-leave-to /* .list-leave-active below version 2.1.8 */ {
  opacity: 0;
  transform: translateY(30px);
}
  2. Animate when order is changed, v-move, *-move class
    • Vue uses FLIP and FLIP doesn't work with elements set to display:inline. Set it to display: inline-block or place elements in a flex context
    <script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.14.1/lodash.min.js"></script>
<div id="flip-list-demo" class="demo">
  <button v-on:click="shuffle">Shuffle</button>
  <transition-group name="flip-list" tag="ul">
    <li v-for="item in items" v-bind:key="item">
  {{ item }}
    </li>
  </transition-group>
</div>

    new Vue({
  el: '#flip-list-demo',
  data: {
    items: [1,2,3,4,5,6,7,8,9]
  },
  methods: {
    shuffle: function () {
  this.items = _.shuffle(this.items)
    }
  }
})

    .flip-list-move {
    transition: transform 1s;
}
  3. JavaScript event hooks
    <script src="https://cdnjs.cloudflare.com/ajax/libs/velocity/1.2.3/velocity.min.js"></script>
<div id="staggered-list-demo">
  <input v-model="query">
  <transition-group
    name="staggered-fade"
    tag="ul"
    v-bind:css="false"
    v-on:before-enter="beforeEnter"
    v-on:enter="enter"
    v-on:leave="leave"
  >
    <li
  v-for="(item, index) in computedList"
  v-bind:key="item.msg"
  v-bind:data-index="index"
    >{{ item.msg }}</li>
  </transition-group>
</div>

    new Vue({
  el: '#staggered-list-demo',
  data: {
    query: '',
    list: [
  { msg: 'Bruce Lee' },
  { msg: 'Jackie Chan' },
  { msg: 'Chuck Norris' },
  { msg: 'Jet Li' },
  { msg: 'Kung Fury' }
    ]
  },
  computed: {
    computedList: function () {
  var vm = this
  return this.list.filter(function (item) {
    return item.msg.toLowerCase().indexOf(vm.query.toLowerCase()) !== -1
  })
    }
  },
  methods: {
    beforeEnter: function (el) {
  el.style.opacity = 0
  el.style.height = 0
    },
    enter: function (el, done) {
  var delay = el.dataset.index * 150
  setTimeout(function () {
    Velocity(
      el,
      { opacity: 1, height: '1.6em' },
      { complete: done }
    )
  }, delay)
    },
    leave: function (el, done) {
  var delay = el.dataset.index * 150
  setTimeout(function () {
    Velocity(
      el,
      { opacity: 0, height: 0 },
      { complete: done }
    )
  }, delay)
    }
  }
})
9.1.8.3. Make transition reusable
Vue.component('my-special-transition', {
  template: '\
    <transition\
      name="very-special-transition"\
      mode="out-in"\
      v-on:before-enter="beforeEnter"\
      v-on:after-enter="afterEnter"\
    >\
      <slot></slot>\
    </transition>\
  ',
  methods: {
    beforeEnter: function (el) {
      // ...
    },
    afterEnter: function (el) {
      // ...
    }
  }
})

Functional component 

Vue.component('my-special-transition', {
  functional: true,
  render: function (createElement, context) {
    var data = {
      props: {
        name: 'very-special-transition',
        mode: 'out-in'
      },
      on: {
        beforeEnter: function (el) {
          // ...
        },
        afterEnter: function (el) {
          // ...
        }
      }
    }
    return createElement('transition', data, context.children)
  }
})
9.1.8.4. State Transition

Achieve transition by changing state (e.g. prop) of data.

  1. Watchers, Tween.js, Color.js
    <script src="https://cdn.jsdelivr.net/npm/tween.js@16.3.4"></script>
<div id="animated-number-demo">
  <input v-model.number="number" type="number" step="20">
  <p>{{ animatedNumber }}</p>
</div>

new Vue({
  el: '#animated-number-demo',
  data: {
    number: 0,
    animatedNumber: 0
  },
  watch: {
    number: function(newValue, oldValue) {
      var vm = this
      function animate () {
        if (TWEEN.update()) {
          requestAnimationFrame(animate)
        }
      }
      new TWEEN.Tween({ tweeningNumber: oldValue })
        .easing(TWEEN.Easing.Quadratic.Out)
        .to({ tweeningNumber: newValue }, 500)
        .onUpdate(function () {
          vm.animatedNumber = this.tweeningNumber.toFixed(0)
        })
        .start()
      animate()
    }
  }
})

9.1.9. Mixins

9.1.9.1. Local vs Global
// define a mixin object
var myMixin = {
  created: function () {
    this.hello()
  },
  methods: {
    hello: function () {
      console.log('hello from mixin!')
    }
  }
}
// define a component that uses this mixin
var Component = Vue.extend({
  mixins: [myMixin]
})
var component = new Component() // => "hello from mixin!"
  • Global mixin

    • Use global mixins sparsely and carefully, because it affects every single Vue instance created, including third party components. In most cases, you should only use it for custom option handling like demonstrated in the example above. It’s also a good idea to ship them as Plugins to avoid duplicate application 

        // inject a handler for `myOption` custom option
  Vue.mixin({
created: function () {
  var myOption = this.$options.myOption
  if (myOption) {
    console.log(myOption)
  }
}
  })
  new Vue({
myOption: 'hello!'
  })
  // => "hello!"
9.1.9.2. Merge option with hook functions

mixin will be called before the component's own hooks. 

var mixin = {
  created: function () {
    console.log('mixin hook called')
  }
}
new Vue({
  mixins: [mixin],
  created: function () {
    console.log('component hook called')
  }
})
// => "mixin hook called"
// => "component hook called"
9.1.9.3. Merge option with Object value

The component's options will take priority when conflict keys exist 

var mixin = {
  methods: {
    foo: function () {
      console.log('foo')
    },
    conflicting: function () {
      console.log('from mixin')
    }
  }
}
var vm = new Vue({
  mixins: [mixin],
  methods: {
    bar: function () {
      console.log('bar')
    },
    conflicting: function () {
      console.log('from self')
    }
  }
})
vm.foo() // => "foo"
vm.bar() // => "bar"
vm.conflicting() // => "from self"
9.1.9.4. Custom Option Merge Strategies
Vue.config.optionMergeStrategies.myOption = function (toVal, fromVal) {
  // return mergedVal
}

var strategies = Vue.config.optionMergeStrategies
strategies.myOption = strategies.methods // the default strategy for most object-based options

// more advanced
const merge = Vue.config.optionMergeStrategies.computed
Vue.config.optionMergeStrategies.vuex = function (toVal, fromVal) {
  if (!toVal) return fromVal
  if (!fromVal) return toVal
  return {
    getters: merge(toVal.getters, fromVal.getters),
    state: merge(toVal.state, fromVal.state),
    actions: merge(toVal.actions, fromVal.actions)
  }
}

9.1.10. Custom Directive

9.1.10.1. Basics
<input v-focus>

// Register a global custom directive called v-focus
Vue.directive('focus', {
  // When the bound element is inserted into the DOM...
  inserted: function (el) {
    // Focus the element
    el.focus()
  }
})

Register it locally for a component, using the directives option inside the component 

directives: {
  focus: {
    // directive definition
    inserted: function (el) {
      el.focus()
    }
  }
}
9.1.10.2. Option - Hook Functions

Other than inserted, there're other optional hooks

bind
called only once, when the directive is first bound to the element. This is where you can do one-time setup work.
inserted
called when the bound element has been inserted into its parent node (this only guarantees parent node presence, not necessarily in-document).
update
called after the containing component’s VNode has updated, but possibly before its children have updated. The directive’s value may or may not have changed, but you can skip unnecessary updates by comparing the binding’s current and old values (see below on hook arguments).
componentUpdated
called after the containing component’s VNode and the VNodes of its children have updated.

unbind: called only once, when the directive is unbound from the element.

Shorthand for same behavior on bind and update 

Vue.directive('color-swatch', function (el, binding) {
  el.style.backgroundColor = binding.value
})
  1. Hooks are passed these arguments

    Apart from el, you should treat these arguments as read-only and never modify them. If you need to share information across hooks, it is recommended to do so through element’s dataset.

    el
    The element the directive is bound to. This can be used to directly manipulate the DOM.
    binding
    An object containing the following properties.
    name
    The name of the directive, without the v- prefix.
    value
    The value passed to the directive. For example in v-my-directive="1 + 1", the value would be 2.
    oldValue
    The previous value, only available in update and componentUpdated. It is available whether or not the value has changed.
    expression
    The expression of the binding as a string. For example in v-my-directive="1 + 1", the expression would be "1 + 1".
    arg
    The argument passed to the directive, if any. For example in v-my-directive:foo, the arg would be "foo".
    modifiers
    An object containing modifiers, if any. For example in v-my-directive.foo.bar, the modifiers object would be { foo: true, bar: true }
    vnode
    The virtual node produced by Vue’s compiler. See the VNode API for full details.
    oldVnode
    The previous virtual node, only available in the update and componentUpdated hooks.
9.1.10.3. Example
<div id="hook-arguments-example" v-demo:foo.a.b="message"></div>

Vue.directive('demo', {
  bind: function (el, binding, vnode) {
    var s = JSON.stringify
    el.innerHTML =
      'name: '       + s(binding.name) + '<br>' +
      'value: '      + s(binding.value) + '<br>' +
      'expression: ' + s(binding.expression) + '<br>' +
      'argument: '   + s(binding.arg) + '<br>' +
      'modifiers: '  + s(binding.modifiers) + '<br>' +
      'vnode keys: ' + Object.keys(vnode).join(', ')
  }
})
new Vue({
  el: '#hook-arguments-example',
  data: {
    message: 'hello!'
  }
})

// result
name: "demo"
value: "hello!"
expression: "message"
argument: "foo"
modifiers: {"a":true,"b":true}
vnode keys: tag, data, children, text, elm, ns, context, functionalContext, functionalOptions, functionalScopeId, key, componentOptions, componentInstance, parent, raw, isStatic, isRootInsert, isComment, isCloned, isOnce, asyncFactory, asyncMeta, isAsyncPlaceholder

Pass multiple values to directive 

<div v-demo="{ color: 'white', text: 'hello!' }"></div>

Vue.directive('demo', function (el, binding) {
  console.log(binding.value.color) // => "white"
  console.log(binding.value.text)  // => "hello!"
})

9.1.11. Plugin vue:plugin

Third party plugins and libraries https://github.com/vuejs/awesome-vue#components--libraries

9.1.11.1. Basics

Plugins usually add global-level functionality to Vue. There is no strictly defined scope for a plugin - there are typically several types of plugins you can write:

  • Add some global methods or properties. e.g. vue-custom-element
  • Add one or more global assets: directives/filters/transitions etc. e.g. vue-touch
  • Add some component options by global mixin. e.g. vue-router
  • Add some Vue instance methods by attaching them to Vue.prototype.
  • A library that provides an API of its own, while at the same time injecting some combination of the above. e.g. vue-router

Official Vue.js plugins such as vue-router automatically calls Vue.use() if vue is available as a global variable. 

MyPlugin.install = function (Vue, options) {
  // 1. add global method or property
  Vue.myGlobalMethod = function () {
    // something logic ...
  }
  // 2. add a global asset
  Vue.directive('my-directive', {
    bind (el, binding, vnode, oldVnode) {
      // something logic ...
    }
    ...
  })
  // 3. inject some component options
  Vue.mixin({
    created: function () {
      // something logic ...
    }
    ...
  })
  // 4. add an instance method
  Vue.prototype.$myMethod = function (methodOptions) {
    // something logic ...
  }
}

Use a Plugin 

// calls `MyPlugin.install(Vue)`
Vue.use(MyPlugin)

Vue.use(MyPlugin, { someOption: true })

// When using CommonJS via Browserify or Webpack
var Vue = require('vue')

var VueRouter = require('vue-router')
// Don't forget to call this
Vue.use(VueRouter)
9.1.11.2. vuex Namespace in Plugin

vue:plugin:vuex:namespace in plugin

9.1.12. Production Development

Without build tools, directly inject javascript Prod :: https://cdn.jsdelivr.net/npm/vue Dev :: https://unpkg.com/vue

9.1.13. vue-router vue:plugin:router vue:router

VueRouter:option:x => option:x

9.1.13.1. <router-view>, <router-link>, option:routes, this.$router, this.$route
<script src="https://unpkg.com/vue/dist/vue.js"></script>
<script src="https://unpkg.com/vue-router/dist/vue-router.js"></script>

<div id="app">
  <h1>Hello App!</h1>
  <p>
    <!-- use router-link component for navigation. -->
    <!-- specify the link by passing the `to` prop. -->
    <!-- `<router-link>` will be rendered as an `<a>` tag by default -->
    <!-- it gets .router-link-active class when the route is matched -->
    <router-link to="/foo">Go to Foo</router-link>
    <router-link to="/bar">Go to Bar</router-link>
  </p>
  <!-- route outlet -->
  <!-- component matched by the route will render here -->
  <router-view></router-view>
</div>

// 0. If using a module system (e.g. via vue-cli), import Vue and VueRouter
// and then call `Vue.use(VueRouter)`.

// 1. Define route components.
// These can be imported from other files
const Foo = { template: '<div>foo</div>' }
const Bar = { template: '<div>bar</div>' }

// 2. Define some routes
// Each route should map to a component. The "component" can
// either be an actual component constructor created via
// `Vue.extend()`, or just a component options object.
// We'll talk about nested routes later.
const routes = [
  { path: '/foo', component: Foo },
  { path: '/bar', component: Bar }
]

// 3. Create the router instance and pass the `routes` option
// You can pass in additional options here, but let's
// keep it simple for now.
const router = new VueRouter({
  routes // short for `routes: routes`
})

// 4. Create and mount the root instance.
// Make sure to inject the router with the router option to make the
// whole app router-aware.
const app = new Vue({
  router
}).$mount('#app')

// Now the app has started!

In any component, this.$router vue:router:router and this.$route vue:router:route are available 

// Home.vue
export default {
  computed: {
    username () {
      // We will see what `params` is shortly
      return this.$route.params.username
    }
  },
  methods: {
    goBack () {
      window.history.length > 1
        ? this.$router.go(-1)
        : this.$router.push('/')
    }
  }
}
9.1.13.2. option:routes, $route.params, regex, option:routes[]:name, option:routes[]:component
const User = {
  template: '<div>User {{ $route.params.id }}</div>'
}

const router = new VueRouter({
  routes: [
    // dynamic segments start with a colon
    { path: '/user/:id', component: User }
  ]
})
  1. regex pattern in option:routes

    vue-router uses path-to-regexp whihc is used by Express as well. The earlier a route is defined, the higher priority it gets. 

    import Vue from 'vue'
import VueRouter from 'vue-router'

Vue.use(VueRouter)

// The matching uses path-to-regexp, which is the matching engine used
// by express as well, so the same matching rules apply.
// For detailed rules, see https://github.com/pillarjs/path-to-regexp
const router = new VueRouter({
  mode: 'history',
  base: __dirname,
  routes: [
    { path: '/' },
    // params are denoted with a colon ":"
    { path: '/params/:foo/:bar' },
    // a param can be made optional by adding "?"
    { path: '/optional-params/:foo?' },
    // a param can be followed by a regex pattern in parens
    // this route will only be matched if :id is all numbers
    { path: '/params-with-regex/:id(\\d+)' },
    // asterisk can match anything
    { path: '/asterisk/*' },
    // make part of the path optional by wrapping with parens and add "?"
    { path: '/optional-group/(foo/)?bar' }
  ]
})

new Vue({
  router,
  template: `
    <div id="app">
      <h1>Route Matching</h1>
      <ul>
        <li><router-link to="/">/</router-link></li>
        <li><router-link to="/params/foo/bar">/params/foo/bar</router-link></li>
        <li><router-link to="/optional-params">/optional-params</router-link></li>
        <li><router-link to="/optional-params/foo">/optional-params/foo</router-link></li>
        <li><router-link to="/params-with-regex/123">/params-with-regex/123</router-link></li>
        <li><router-link to="/params-with-regex/abc">/params-with-regex/abc</router-link></li>
        <li><router-link to="/asterisk/foo">/asterisk/foo</router-link></li>
        <li><router-link to="/asterisk/foo/bar">/asterisk/foo/bar</router-link></li>
        <li><router-link to="/optional-group/bar">/optional-group/bar</router-link></li>
        <li><router-link to="/optional-group/foo/bar">/optional-group/foo/bar</router-link></li>
      </ul>
      <p>Route context</p>
      <pre>{{ JSON.stringify($route, null, 2) }}</pre>
    </div>
  `
}).$mount('#app')
  2. Named routes, option:routes[]:name
    const router = new VueRouter({
  routes: [
    {
      path: '/user/:userId',
      name: 'user',
      component: User
    }
  ]
})

<router-link :to="{ name: 'user', params: { userId: 123 }}">User</router-link>
router.push({ name: 'user', params: { userId: 123 }})
9.1.13.3. option:routes[]:components, multiple <router-view>'s
<router-view class="view one"></router-view>
<router-view class="view two" name="a"></router-view>
<router-view class="view three" name="b"></router-view>

const router = new VueRouter({
  routes: [
    {
      path: '/',
      components: {
        default: Foo,
        a: Bar,
        b: Baz
      }
    }
  ]
})
9.1.13.4. option:routes[]:redirect, option:routes[]:alias

A redirect means when the user visits /a, and URL will be replaced by /b, and then matched as /b. 

const router = new VueRouter({
  routes: [
    { path: '/a', redirect: '/b' }
  ]
})

const router = new VueRouter({
  routes: [
    { path: '/a', redirect: { name: 'foo' }}
  ]
})

const router = new VueRouter({
  routes: [
    { path: '/a', redirect: to => {
      // the function receives the target route as the argument
      // return redirect path/location here.
    }}
  ]
})

An alias of /a as /b means when the user visits /b, the URL remains /b, but it will be matched as if the user is visiting /a. 

const router = new VueRouter({
  routes: [
    { path: '/a', component: A, alias: '/b' }
  ]
})
9.1.13.5. Route object, $route, this.$route, option:scrollBehavior vue:router:route
  • https://router.vuejs.org/en/api/route-object.html

  • $route => this.$route, router.match(location) 

    router.beforeEach((to, from, next) => {
  // `to` and `from` are both route objects
})

const router = new VueRouter({
  scrollBehavior (to, from, savedPosition) {
// `to` and `from` are both route objects
  }
})
  • absolute path e.g. /foo/bar
  • {} or key/value pairs of dynamic segments and star segments
  • {} or key/vale pairs of the query string /foo?user=1 $route.query.user == 1
  • empty string or anything with #
  • string query and hash
  • e.g. /foo/bar, then .matched is an array of objects in parent to child order.
9.1.13.6. option:scrollBehavior
const router = new VueRouter({
  routes: [...],
  scrollBehavior (to, from, savedPosition) {
    // return desired position
  }
})

savedPosition, is only available if this is a popstate navigation (triggered by the browser's back/forward buttons).

Should return either one of these

  • { x: number, y: number }
  • { selector: string, offset? : { x: number, y: number }}

Scroll to top for all route navigations 

scrollBehavior (to, from, savedPosition) {
  return { x: 0, y: 0 }
}

scrollBehavior (to, from, savedPosition) {
  if (savedPosition) {
    return savedPosition
  } else {
    return { x: 0, y: 0 }
  }
}

// scroll to anchor
scrollBehavior (to, from, savedPosition) {
  if (to.hash) {
    return {
      selector: to.hash
      // , offset: { x: 0, y: 10 }
    }
  }
}
9.1.13.7. option:routes[]:props, component:props

Decouple $route in component. Before 

const User = {
  template: '<div>User {{ $route.params.id }}</div>'
}
const router = new VueRouter({
  routes: [
    { path: '/user/:id', component: User }
  ]
})

Now 

const User = {
  props: ['id'],
  template: '<div>User {{ id }}</div>'
}
const router = new VueRouter({
  routes: [
    { path: '/user/:id', component: User, props: true }

    // for routes with named views, you have to define the `props` option for each named view:
    {
      path: '/user/:id',
      components: { default: User, sidebar: Sidebar },
      props: { default: true, sidebar: false }
    }
  ]
})

option:routes[]:props is set to true, the route.params will be set as the component props.

props is an object 

const router = new VueRouter({
  routes: [
    { path: '/hello/:name', component: Hello, props: true }, // Pass route.params to props
    { path: '/promotion/from-newsletter', component: Promotion, props: { newsletterPopup: false } },
    { path: '/static', component: Hello, props: { name: 'world' }}, // static values
  ]
})

props is a function The URL /search?q=vue would pass {query: 'vue'} as props to the SearchUser component. 

function dynamicPropsFn (route) {
  const now = new Date()
  return {
    name: (now.getFullYear() + parseInt(route.params.years)) + '!'
  }
}

const router = new VueRouter({
  routes: [
    { path: '/search', component: SearchUser, props: (route) => ({ query: route.query.q }) }
    { path: '/dynamic/:years', component: Hello, props: dynamicPropsFn }, // custom logic for mapping between route and props
  ]
})
9.1.13.8. React to params changes, component:watch, component:beforeRouteUpdate

vue:router:react vue:router:component:beforeRouteUpdate vue:router:component:watch params or query chnages won't trigger enter/leave navigation guards and hence this is needed to react to changes. 

const User = {
  template: '...',
  watch: {
    '$route' (to, from) {
      // react to route changes...
    }
  }
}

const User = {
  template: '...',
  beforeRouteUpdate (to, from, next) {
    // react to route changes...
    // don't forget to call next()
  }
}
9.1.13.9. Nested routes, <router-view>, option:routes:children

/user/foo/profile (component:user and component:profile) and /user/foo/posts (component:user and component:posts) 

<div id="app">
  <router-view></router-view>
</div>

const User = {
  template: `
    <div>User {{ $route.params.id }}</div>
    <router-view></router-view>
  `
}

const router = new VueRouter({
  routes: [
    { path: '/user/:id', component: User,
      children: [
        // UserHome will be rendered inside User's <router-view>
        // when /user/:id is matched
        { path: '', component: UserHome },

        {
          // UserProfile will be rendered inside User's <router-view>
          // when /user/:id/profile is matched
          path: 'profile',
          component: UserProfile
        },

        {
          // UserPosts will be rendered inside User's <router-view>
          // when /user/:id/posts is matched
          path: 'posts',
          component: UserPosts
        }
      ]
    }
  ]
})
  1. option:routes:children[]:meta
    const router = new VueRouter({
  routes: [
    {
      path: '/foo',
      component: Foo,
      children: [
        {
          path: 'bar',
          component: Bar,
          // a meta field
          meta: { requiresAuth: true }
        }
      ]
    }
  ]
})

    router.beforeEach((to, from, next) => {
  if (to.matched.some(record => record.meta.requiresAuth)) {
    // this route requires auth, check if logged in
    // if not, redirect to login page.
    if (!auth.loggedIn()) {
      next({
        path: '/login',
        query: { redirect: to.fullPath }
      })
    } else {
      next()
    }
  } else {
    next() // make sure to always call next()!
  }
})
9.1.13.10. Programmatic Navigation, this.$router, router.*

vue:router:router vue:router:router.push Both router and this.$router can be used in component. this.$router means vue-router is imported in every component. <router-link :to="..."> is equivalent to router.push(...)

params are ignored if a path is provided. 

// literal string path
router.push('home')

// object
router.push({ path: 'home' })

// named route
router.push({ name: 'user', params: { userId: 123 }})

// with query, resulting in /register?plan=private
router.push({ path: 'register', query: { plan: 'private' }})

const userId = 123
router.push({ name: 'user', params: { userId }}) // -> /user/123
router.push({ path: `/user/${userId}` }) // -> /user/123
// This will NOT work
router.push({ path: '/user', params: { userId }}) // -> /user
  1. router.push, router.replace, router.push.onComplete, router.push.onAbort

    Callbacks as the 2nd and 3rd arguments. onComplete :: after all async hooks are resolved onAbort :: navigated to the same route, or to a different route before current navigation has finished.

    If destination is the same as the current route and only params are changing (/users/1 -> /users/2), use beforeRouteUpdate to react. vue:router:react

    router.push(location, onComplete?, onAbort?) router.replace(location, onComplete?, onAbort?) :: without pushing a new history entry

  2. router.go(n)

    Similar to window.history.go(n) 

    // go forward by one record, the same as history.forward()
router.go(1)

// go back by one record, the same as history.back()
router.go(-1)

// go forward by 3 records
router.go(3)

// fails silently if there aren't that many records.
router.go(-100)
router.go(100)
9.1.13.11. option:mode

Default is hash mode, change to history mode to change URL without a page reload. 

const router = new VueRouter({
  mode: 'history',
  routes: [
    // some path
    // catch 404
    { path: '*', component: NotFoundComponent }
  ]
})

Apache 

<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteBase /
  RewriteRule ^index\.html$ - [L]
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteRule . /index.html [L]
</IfModule>

nginx 

location / {
  try_files $uri $uri/ /index.html;
}
9.1.13.12. Guards
  • Navigation triggered.
  • Call leave guards in deactivated components.
  • Call global beforeEach guards.
  • Call beforeRouteUpdate guards in reused components (2.2+).
  • Call beforeEnter in route configs.
  • Resolve async route components.
  • Call beforeRouteEnter in activated components.
  • Call global beforeResolve guards (2.5+).
  • Navigation confirmed.
  • Call global afterEach hooks.
  • DOM updates triggered.
  • Call callbacks passed to next in beforeRouteEnter guards with instantiated instances.
  1. global, router.beforeEach, router.beforeResolve, router.afterEach

    Guards/hooks may be resolved asynchronously, the navigation is considered pending before all hooks have been resolved (confirmed). vue:router:router.beforeEach 

    const router = new VueRouter({ ... })

router.beforeEach((to, from, next) => {
  // ...
})

    Always call next function to resolve.

    next() :: move on to the next hook in the pipeline. next(false) :: abort the current navigation. If the URL was changed (either manually by the user or via back button), it will be reset to that of the from route. next('') or next({ path: '' }) :: redirect to a different location. The current navigation will be aborted and a new one will be started. A location object can be passed. Refer to vue:router:router.push next(error) :: navigation will be aborted and the error will be passed to callbacks registered via router.onError().

    router.beforeResolve is similar to router.beforeEach with the difference that resolve guards will be called right before the navigation is confirmed, after all in-component guards and async route components are resolved.

    router.afterEach do not get a next function and cannot affect the navigation 

    router.afterEach((to, from) => {
  // ...
})
  2. per-route, option:routes[]:beforeEnter

    Same as vue:router:router.beforeEach 

    const router = new VueRouter({
  routes: [
    {
      path: '/foo',
      component: Foo,
      beforeEnter: (to, from, next) => {
        // ...
      }
    }
  ]
})
  3. in-component, component:beforeRouteEnter, component:beforeRouteUpdate, component:beforeRouteLeave
    const Foo = {
  template: `...`,
  beforeRouteEnter (to, from, next) {
    // called before the route that renders this component is confirmed.
    // does NOT have access to `this` component instance,
    // because it has not been created yet when this guard is called!

    // you can access to component instance via `vm`

    next(vm => {
      // it will be called when the navigation is confirmed
      // access to component instance via `vm`
    })
  },
  beforeRouteUpdate (to, from, next) {
    // called when the route that renders this component has changed,
    // but this component is reused in the new route.
    // For example, for a route with dynamic params `/foo/:id`, when we
    // navigate between `/foo/1` and `/foo/2`, the same `Foo` component instance
    // will be reused, and this hook will be called when that happens.
    // has access to `this` component instance.
  },
  beforeRouteLeave (to, from, next) {
    // called when the route that renders this component is about to
    // be navigated away from.
    // has access to `this` component instance.

    const answer = window.confirm('Do you really want to leave? you have unsaved changes!')
    if (answer) {
      next()
    } else {
      next(false)
    }    
  }
}
9.1.13.13. Fetch data

Fetch after navigation 

<template>
  <div class="post">
    <div class="loading" v-if="loading">
      Loading...
    </div>

    <div v-if="error" class="error">
      {{ error }}
    </div>

    <div v-if="post" class="content">
      <h2>{{ post.title }}</h2>
      <p>{{ post.body }}</p>
    </div>
  </div>
</template>
export default {
  data () {
    return {
      loading: false,
      post: null,
      error: null
    }
  },
  created () {
    // fetch the data when the view is created and the data is
    // already being observed
    this.fetchData()
  },
  watch: {
    // call again the method if the route changes
    '$route': 'fetchData'
  },
  methods: {
    fetchData () {
      this.error = this.post = null
      this.loading = true
      // replace `getPost` with your data fetching util / API wrapper
      getPost(this.$route.params.id, (err, post) => {
        this.loading = false
        if (err) {
          this.error = err.toString()
        } else {
          this.post = post
        }
      })
    }
  }
}

Fetch before navigation 

export default {
  data () {
    return {
      post: null,
      error: null
    }
  },
  beforeRouteEnter (to, from, next) {
    getPost(to.params.id, (err, post) => {
      next(vm => vm.setData(err, post))
    })
  },
  // when route changes and this component is already rendered,
  // the logic will be slightly different.
  beforeRouteUpdate (to, from, next) {
    this.post = null
    getPost(to.params.id, (err, post) => {
      this.setData(err, post)
      next()
    })
  },
  methods: {
    setData (err, post) {
      if (err) {
        this.error = err.toString()
      } else {
        this.post = post
      }
    }
  }
}
9.1.13.14. Lazy loading

Only load components when they are triggered by navigation 

const Foo = () => Promise.resolve({ /* component definition */ })
import('./Foo.vue') // returns a Promise

Webpack 

const Foo = () => import('./Foo.vue')

// everything is the same in router config
const router = new VueRouter({
  routes: [
    { path: '/foo', component: Foo }
  ]
})

Group multiple components and lazy load them 

const Foo = () => import(/* webpackChunkName: "group-foo" */ './Foo.vue')
const Bar = () => import(/* webpackChunkName: "group-foo" */ './Bar.vue')
const Baz = () => import(/* webpackChunkName: "group-foo" */ './Baz.vue')

9.1.14. State Management, Vuex

vue:state management vue:plugin:vuex

9.1.14.1. Basics, option:state

vuex is a vue:plugin

Vuex Doc

npm install vuex –save

./src/store/store.js (or index.js) 

import Vue from 'vue'
import Vuex from 'vuex'

Vue.use(Vuex)

export default new Vuex.Store({
  state: {
    count: 0
  },
  mutations: {
    increment: state => state.count++, // shorthand for
    // increment (state) { state.count++ } 
    decrement: state => state.count--
  },
  actions: {
    // actions can be defined directly here to so that
    // they can interact with other actions/modules
  }
})

./src/main.js 

import Vue from 'vue'
import App from './App'
import router from './router'
import store from './store/store' // new, if store/index.js is used, use ./store

Vue.config.productionTip = false

/* eslint-disable no-new */
new Vue({
  el: '#app',
  router,
  store, // new
  template: '<App/>',
  components: { App }

})

App.vue 

<template>
  <div id="app">
    <img src="./assets/logo.png">
    <router-view/>
    <p>{{ count }}</p>
    <p>
      <button @click="increment">+</button>
      <button @click="decrement">-</button>
    </p>
  </div>
</template>

<script>

export default {
  name: 'app',
  computed: {
    count () {
      return this.$store.state.count
    }
  },
  methods: {
    increment () {
      this.$store.commit('increment')
    },
    decrement () {
      this.$store.commit('decrement')
    }
  }
}
</script>
9.1.14.2. mapState in component computed props

mapState is to get state as it is without calculation or maybe has to do calculations with local state. It can save some keystrokes: Before App.vue 

<script>
export default {
  name: 'app',
  computed: {
    count () {
      return this.$store.state.count
    }
  },
  // some methods: {}
}
</script>

Use mapState 

// in full builds helpers are exposed as Vuex.mapState
import { mapState } from 'vuex'

export default {
  // ...
  computed: mapState({
    // arrow functions can make the code very succinct!
    count: state => state.count,

    // passing the string value 'count' is same as `state => state.count`
    countAlias: 'count',

    // to access local state with `this`, a normal function must be used
    countPlusLocalState (state) {
      return state.count + this.localCount
    }
  })
}

Use array in mapState to map this.count to store.state.count 

computed: mapState([
  // map this.count to store.state.count
  'count'
])

mapState returns an object inside the component's computed. How to combine with other computed properties? 

computed: {
  localComputed () { /* ... */ },
  // mix this into the outer object with the object spread operator
  ...mapState({
    // ...
  })
}
9.1.14.3. option:getters, mapGetters in component computed props

getters or mapGetters are to get state and compute on it.

./src/store/store.js 

const store = new Vuex.Store({
  state: {
    todos: [
      { id: 1, text: '...', done: true },
      { id: 2, text: '...', done: false }
    ]
  },
  getters: {
    doneTodos: state => {
      return state.todos.filter(todo => todo.done)
    },
    doneTodosCount: (state, getters) => {
      return getters.doneTodos.length
    },
    // pass parameter
    getTodoById: (state, getters) => (id) => {
      return state.todos.find(todo => todo.id === id)
    }
  }
})

store.getters.doneTodos // -> [{ id: 1, text: '...', done: true }]
store.getters.doneTodosCount // -> 1
store.getters.getTodoById(2) // -> { id: 2, text: '...', done: false }

In component 

computed: {
  doneTodosCount () {
    return this.$store.getters.doneTodosCount
  }
}

mapGetters in component computed props 

import { mapGetters } from 'vuex'

export default {
  // ...
  computed: {
    // mix the getters into computed with object spread operator
    ...mapGetters([
      'doneTodosCount',
      'anotherGetter',
      // ...
    ])
  }
}

Change name 

...mapGetters({
  // map `this.doneCount` to `store.getters.doneTodosCount`
  doneCount: 'doneTodosCount'
})
9.1.14.4. option:mutations, commit/mapMutations in component, Synchronous vue:plugin:vuex:mutations

Mutation is the only way to actually change state

Mutation is defined in Vuex and it has to be synchronous!

Add new prop to state 

Vue.set(obj, 'newProp', 123)

// or rebuild the whole state
state.obj = { ...state.obj, newProp: 123 }
  1. Mutation in store, commit in component
    export default new Vuex.Store({
  state: {
    count: 1
  },
  mutations: {
    increment: state => state.count++, // shorthand for
    // increment (state) { state.count++ } 
  }
})

    commit in component 

    methods: {
  increment () {
    this.$store.commit('increment')
  },
  decrement () {
    this.$store.commit('decrement')
  }
}

    Mutation-Commit with payload 

    mutations: {
  increment (state, payload) {
    state.count += payload.amount
  }
}

// in component
methods: {
  increment: () {
    this.$store.commit('increment', {
      amount: 10
    })
    // object commit form
    /*
    this.$store.commit({
      type: 'increment',
      amound: 10
    })
    */
  }
}
  2. mapMutations in component

    save typing in component without typing commit 

    import { mapMutations } from 'vuex'

export default {
  // ...
  methods: {
    ...mapMutations([
      'increment', // map `this.increment()` to `this.$store.commit('increment')`

      // `mapMutations` also supports payloads:
      'incrementBy' // map `this.incrementBy(amount)` to `this.$store.commit('incrementBy', amount)`
    ]),
    ...mapMutations({
      add: 'increment' // map `this.add()` to `this.$store.commit('increment')`
    })
  }
}
  3. Constant for Mutation
    // mutation-types.js
export const SOME_MUTATION = 'SOME_MUTATION'

// store.js
import Vuex from 'vuex'
import { SOME_MUTATION } from './mutation-types'

const store = new Vuex.Store({
  state: { ... },
  mutations: {
    // we can use the ES2015 computed property name feature
    // to use a constant as the function name
    [SOME_MUTATION] (state) {
      // mutate state
    }
  }
})
9.1.14.5. option:actions with context, dispatch/mapActions in component, Asynchronous

Actions commit mutations

  1. Basics
    const store = new Vuex.Store({
  state: {
    count: 0
  },
  mutations: {
    increment (state) {
      state.count++
    }
  },
  actions: {
    increment (context) {
      // context exposes the same set of methods/props on the store instance
      // e.g. context.state, context.getters
      // but context is not the store instance
      context.commit('increment')
    }

    /* the above can be shortened to

    increment ({ commit }) {
      commit('increment')
    }
    */

  }
})

    In component 

    this.$store.dispatch('increment')
  2. mapActions and Asynchronous

    store.js 

    actions: {
  // checkout only needs commit and state from context
  // products is a payload
  checkout ({ commit, state }, products) {

    // save the items currently in the cart
    const savedCartItems = [...state.cart.added]

    // send out checkout request, and optimistically
    // clear the cart
    commit(types.CHECKOUT_REQUEST)

    // the shop API accepts a success callback and a failure callback
    shop.buyProducts(

      products,

      // handle success
      () => commit(types.CHECKOUT_SUCCESS),

      // handle failure
      () => commit(types.CHECKOUT_FAILURE, savedCartItems)

    )
  }
}

    In component 

    import { mapActions } from 'vuex'

export default {
  // ...
  methods: {
    ...mapActions([
      'increment', // map `this.increment()` to `this.$store.dispatch('increment')`

      // `mapActions` also supports payloads:
      'incrementBy' // map `this.incrementBy(amount)` to `this.$store.dispatch('incrementBy', amount)`
    ]),
    ...mapActions({
      add: 'increment' // map `this.add()` to `this.$store.dispatch('increment')`
    })
  }
}
  3. Series of actions, return Promise
    actions: {
  actionA ({ commit }) {
    return new Promise((resolve, reject) => {
      setTimeout(() => {
        commit('someMutation')
        resolve()
      }, 1000)
    })
  },
  actionB ({ dispatch, commit }) {
    return dispatch('actionA').then(() => {
      commit('someOtherMutation')
    })
  }
}

    store.dispatch('actionA').then(() => {
  // ...
})
  4. Series of actions, async/await

    // assuming `getData()` and `getOtherData()` return Promises

    actions: { async actionA ({ commit }) { commit('gotData', await getData()) }, async actionB ({ dispatch, commit }) { await dispatch('actionA') // wait for `actionA` to finish commit('gotOtherData', await getOtherData()) } }

9.1.14.6. option:modules
const moduleA = {
  state: { ... },
  mutations: { ... },
  actions: { ... },
  getters: { ... }
}

const moduleB = {
  state: { ... },
  mutations: { ... },
  actions: { ... }
}

const store = new Vuex.Store({
  modules: {
    a: moduleA,
    b: moduleB
  }
})

store.state.a // -> `moduleA`'s state
store.state.b // -> `moduleB`'s state

Register a module after the store has been created 

// register a module `myModule`
store.registerModule('myModule', {
  // ...
})

// register a nested module `nested/myModule`
store.registerModule(['nested', 'myModule'], {
  // ...
})

store.unregisterModule(moduleName)
  1. Module local state and rootState
    const moduleA = {
  state: { count: 0 },
  mutations: {
    increment (state) {
      // `state` is the local module state
      state.count++
    }
  },

  getters: {
    doubleCount (state, rootState) {
      // rootState is root state
      return state.count * 2 + rootState.count
    }
  }

  actions: {
    incrementIfOddOnRootSum ({ state, commit, rootState }) {
      // local state
      if ((state.count + rootState.count) % 2 === 1) {
        commit('increment')
      }
    }
  }
}
  2. Namespace

    Namespaced getters and actions will receive localized getters, dispatch and commit. In other words, you can use the module assets without writing prefix in the same module. 

    const store = new Vuex.Store({
  modules: {
    account: {
      namespaced: true,

      // module assets
      state: { ... }, // module state is already nested and not affected by namespace option
      getters: {
        isAdmin () { ... } // -> getters['account/isAdmin']
      },
      actions: {
        login () { ... } // -> dispatch('account/login')
      },
      mutations: {
        login () { ... } // -> commit('account/login')
      },

      // nested modules
      modules: {
        // inherits the namespace from parent module
        myPage: {
          state: { ... },
          getters: {
            profile () { ... } // -> getters['account/profile']
          }
        },

        // further nest the namespace
        posts: {
          namespaced: true,

          state: { ... },
          getters: {
            popular () { ... } // -> getters['account/posts/popular']
          }
        }
      }
    }
  }
})

    In component 

    computed: {
  ...mapState({
    a: state => state.some.nested.module.a,
    b: state => state.some.nested.module.b
  })
},
methods: {
  ...mapActions([
    'some/nested/module/foo',
    'some/nested/module/bar'
  ])
}

    Or this 

    computed: {
  ...mapState('some/nested/module', {
    a: state => state.a,
    b: state => state.b
  })
},
methods: {
  ...mapActions('some/nested/module', [
    'foo',
    'bar'
  ])
}

    Or 

    import { createNamespacedHelpers } from 'vuex'

const { mapState, mapActions } = createNamespacedHelpers('some/nested/module')

export default {
  computed: {
    // look up in `some/nested/module`
    ...mapState({
      a: state => state.a,
      b: state => state.b
    })
  },
  methods: {
    // look up in `some/nested/module`
    ...mapActions([
      'foo',
      'bar'
    ])
  }
}
  3. rootGetters
    modules: {
  foo: {
    namespaced: true,

    getters: {
      // `getters` is localized to this module's getters
      // you can use rootGetters via 4th argument of getters
      someGetter (state, getters, rootState, rootGetters) {
        getters.someOtherGetter // -> 'foo/someOtherGetter'
        rootGetters.someOtherGetter // -> 'someOtherGetter'
      },
      someOtherGetter: state => { ... }
    },

    actions: {
      // dispatch and commit are also localized for this module
      // they will accept `root` option for the root dispatch/commit
      someAction ({ dispatch, commit, getters, rootGetters }) {
        getters.someGetter // -> 'foo/someGetter'
        rootGetters.someGetter // -> 'someGetter'

        dispatch('someOtherAction') // -> 'foo/someOtherAction'
        dispatch('someOtherAction', null, { root: true }) // -> 'someOtherAction'

        commit('someMutation') // -> 'foo/someMutation'
        commit('someMutation', null, { root: true }) // -> 'someMutation'
      },
      someOtherAction (ctx, payload) { ... }
    }
  }
}
  4. Namespace in Plugin Development vue:plugin:vuex:namespace in plugin
    // get namespace value via plugin option
// and returns Vuex plugin function
export function createPlugin (options = {}) {
  return function (store) {
    // add namespace to plugin module's types
    const namespace = options.namespace || ''
    store.dispatch(namespace + 'pluginAction')
  }
}
9.1.14.7. option:strict, strict mode

In strict mode, whenever Vuex state is mutated outside of mutation handlers, an error will be thrown. This ensures that all state mutations can be explicitly tracked by debugging tools. 

const store = new Vuex.Store({
  // ...
  strict: process.env.NODE_ENV !== 'production'
})
9.1.14.8. option:watch

watch(getter: Function, cb: Function, options?: Object)

Reactively watch a getter function's return value, and call the callback when the value changes. The getter receives the store's state as the only argument. Accepts an optional options object that takes the same options as Vue's vm.$watch method.

To stop watching, call the returned handle function.

9.1.14.9. option:plugin

plugins option that exposes hooks for each mutation. A Vuex plugin is simply a function that receives the store as the only argument 

const myPlugin = store => {
  // called when the store is initialized
  store.subscribe((mutation, state) => {
    // called after every mutation.
    // The mutation comes in the format of `{ type, payload }`.
  })
}

const store = new Vuex.Store({
  // ...
  plugins: [myPlugin]
})
9.1.14.10. App Structure, Example

Shopping Cart Example 

├── index.html
├── main.js
├── api
│   └── ... # abstractions for making API requests
├── components
│   ├── App.vue
│   └── ...
└── store
    ├── index.js          # where we assemble modules and export the store
    ├── actions.js        # root actions
    ├── mutations.js      # root mutations
    └── modules
        ├── cart.js       # cart module
        └── products.js   # products module

./api/shop.js 

/**
 * Mocking client-server processing
 */
const _products = [
  {"id": 1, "title": "iPad 4 Mini", "price": 500.01, "inventory": 2},
  {"id": 2, "title": "H&M T-Shirt White", "price": 10.99, "inventory": 10},
  {"id": 3, "title": "Charli XCX - Sucker CD", "price": 19.99, "inventory": 5}
]

export default {
  getProducts (cb) {
    setTimeout(() => cb(_products), 100)
  },

  buyProducts (products, cb, errorCb) {
    setTimeout(() => {
      // simulate random checkout failure.
      (Math.random() > 0.5 || navigator.userAgent.indexOf('PhantomJS') > -1)
        ? cb()
        : errorCb()
    }, 100)
  }
}

./store/index.js 

import Vue from 'vue'
import Vuex from 'vuex'
import * as actions from './actions'
import * as getters from './getters'
import cart from './modules/cart'
import products from './modules/products'
import createLogger from '../../../src/plugins/logger'

Vue.use(Vuex)

const debug = process.env.NODE_ENV !== 'production'

export default new Vuex.Store({
  actions,
  getters,
  modules: {
    cart,
    products
  },
  strict: debug,
  plugins: debug ? [createLogger()] : []
})

./store/modules/products.js 

import shop from '../../api/shop'
import * as types from '../mutation-types'

// initial state
const state = {
  all: []
}

// getters
const getters = {
  allProducts: state => state.all
}

// actions
const actions = {
  getAllProducts ({ commit }) {
    shop.getProducts(products => {
  commit(types.RECEIVE_PRODUCTS, { products })
    })
  }
}

// mutations
const mutations = {
  [types.RECEIVE_PRODUCTS] (state, { products }) {
    state.all = products
  },

  [types.ADD_TO_CART] (state, { id }) {
    state.all.find(p => p.id === id).inventory--
  }
}

export default {
  state,
  getters,
  actions,
  mutations
}

./components/App.vue 

<template>
  <div id="app">
    <h1>Shopping Cart Example</h1>
    <hr>
    <h2>Products</h2>
    <product-list></product-list>
    <hr>
    <cart></cart>
  </div>
</template>

<script>
import ProductList from './ProductList.vue'
import Cart from './Cart.vue'
export default {
  components: { ProductList, Cart }
}
</script>

./component/ProductList.vue 

<template>
  <ul>
    <li v-for="p in products">
      {{ p.title }} - {{ p.price | currency }}
      <br>
      <button
        :disabled="!p.inventory"
        @click="addToCart(p)">
        Add to cart
      </button>
    </li>
  </ul>
</template>

<script>
import { mapGetters, mapActions } from 'vuex'
export default {
  computed: mapGetters({
    products: 'allProducts'
  }),
  methods: mapActions([
    'addToCart'
  ]),
  created () {
    this.$store.dispatch('getAllProducts')
  }
}
</script>

9.1.15. vue-cli vue:vue-cli

9.1.15.1. Basics
# If vue-cli is previously installed, uninstall it first
# npm install -g vue-cli
# npm uninstall vue-cli -g
# or
# yarn global remove vue-cli

# @vue/cli requires NPM 8.11
npm install -g @vue/cli
# yarn global add @vue/cli

# check version. Should be 3.x
vue --version

# create a project in a new dir
vue create hello-world

# create a project inside a dir
vue create .

# Default preset: babel and ESLint
# Manual select features for preset. Then a JSON file `~/.vuerc` is created to store the preset

vue create --help

vue ui

# install a Vue CLI plugin. This resolves @vue/eslint to the full pkg name @vue/cli-plugin-eslint, install it from npm and invokes its generator
vue adde eslint

# same as above
vue add cli-plugin-eslint
9.1.15.2. Services
@vue/cli-service is installed
vue-cli-service
(no term)
scripts in package.json
serve
vue-cli-service serve
9.1.15.3. Typescript
Manual select preset
Babel, TypeScript, CSS Pre-processors, Linter / Formatter
Use class-style component syntax

Yes Classic Vue Syntax vs Class-Style Component Syntax data is a function vs data.message is a property Computed properties become getter #+NAME Classic Vue Syntax 

import Vue, { VNode } from 'vue'

export const HelloComponent = Vue.extend({
data () {
    return {
    message: 'Hello',
    }
},
methods: {
    greet (): string {
    return this.message + ' world';
    }
},
computed: {
    greeting(): string {
    return this.greet() + '!';
    }
},
render (createElement): VNode {
    return createElement('div', this.greeting);
}
});

#+NAME Class-Style Component Syntax 

import Vue from 'vue'
import Component from 'vue-class-component'

@Component({
template: '<div></div>'
})
export default class HelloComponent extends Vue {
message: string = 'Hello'
greet(): string {
    return this.message + ' world';
}
get greeting() {
    return this.greet() + '!';
}
}
(no term)
Use Babel alongside TypeScript for auto-detected polyfills? Yes (default)
Pick a CSS pre-processer (PostCSS, Autoprefixer and CSS Modules are supported by default)
Sass/SCSS (with node-sass)
Pick a linter / formatter config
TSLint
Pick additional lint features
Lint on save
Where do you prefer placing config for Babel, PostCSS, ESLint, etc.?
In dedicated config files. Want to separate them instead of cluttering package.json
  • tslint.json
  • tsconfig.json
  • postcss.config.js
  • babel.config.js
(no term)
Directories
  • public
  • src
    main.ts
    runs first

9.1.16. Dev - Webpack Template

9.1.16.1. Basics

  • Install node module vue:vue-cli 

    vue init webpack my-project
cd my-project
npm install
npm run dev

    Other template can be used 

    # vue init <template-name> <project-name>
vue init webpack-simple my-project

# Template on github
vue init username/repo my-project

# Template on Bitbucket
vue init bitbucket:username/repo my-project

# for a tag or branch
vue init bitbucket:username/repo#branch my-project

# local template
vue init ~/fs/path/to-custom-template my-project

    Webpack Template Doc

9.1.16.2. npm

Default port is 8080 and it can be auto adjusted. It loads the vuejs-templates/webpack npm run dev includes

  • webpack + vue-loader for single file Vue components
  • state preserving hot-reload
  • state preserving compilation error overlay
  • Lint-on-save with ESLint
  • Source maps

npm run build :: build for production

  • UglifyJS
  • html-minifier https://github.com/kangax/html-minifier
  • CSS across all components extracted into a single file and minified with cssnano
  • Static assets compiled with versio hashes for efficient long-term caching and an auto-generated production index.html with proper URLs to these generated assets
  • Use npm run build --report to build with bundle size analytics

npm run unit :: unit tests run in PhantomJS with Karma + Mocha + karma-webpack

  • support ES2015+ in test files
  • support all webpack loaders
  • easy mock injection

num run e2e :: end-to-end tests with Nightwatch

  • run tests in multiple browsers in parallel
  • works with one command out of the box
    • Selenium and chromedrive dependencies automatically handled
    • automatically spawns the Selenium Server
9.1.16.3. Structure
.
├── build/                      # webpack config files for dev and prod. Don't touch unless to customize Webpack loaders
│   └── ...                     # e.g. webpack.base.conf.js
├── config/
│   ├── index.js                # main project config. vue:webpack:integrate backend
│   └── ...
├── src/                        # app files
│   ├── main.js                 # app entry file
│   ├── App.vue                 # main app component
│   ├── components/             # ui components
│   │   └── ...
│   └── assets/                 # module assets (processed by webpack)
│       └── ...
├── static/                     # pure static assets (directly copied and not processed by Webpack)
├── test/
│   └── unit/                   # unit tests
│   │   ├── specs/              # test spec files
│   │   ├── index.js            # test build entry file
│   │   └── karma.conf.js       # test runner config file
│   └── e2e/                    # e2e tests
│   │   ├── specs/              # test spec files
│   │   ├── custom-assertions/  # custom assertions for e2e tests
│   │   ├── runner.js           # test runner script
│   │   └── nightwatch.conf.js  # test runner config file
├── .babelrc                    # babel config
├── .postcssrc.js               # postcss config
├── .eslintrc.js                # eslint config
├── .editorconfig               # editor config
├── index.html                  # index.html template. Webpack will generate assets and auto inject into it during dev and builds.
└── package.json                # build scripts and dependencies
9.1.16.4. Linter configuration

Modify ./.eslintrc.js Add under rule 

"semi": [2, "always"]

Preset standard is set. Checkout all the rules.

9.1.16.5. CSS Pre-processor

This boilerplate has pre-configured LESS, SASS, Stylus adn PostCSS. Just install the appropriate webpack loader. 

npm install sass-loader node-sass --save-dev

After, in *.vue components 

<style lang="scss">
/* write SASS! */
</style>

lang="scss" corresponds to the CSS-superset syntax (with curly braces and semicolons). lang="sass" corresponds to the indentation-based syntax.

Standalone CSS Files Import global and custom style files from your root App.vue component 

<!-- App.vue -->
<style src="./styles/global.less" lang="less"></style>

For other 3rd party, e.g. Bootstrap, place them inside /static and reference them directly in index.html.

9.1.16.6. Processed Static Assets => /src/assets

In *.vue components, all templates and CSS are parsed by vue-html-loader and css-loader to look for asset URLs.

<img src="./logo.png"> and background: url(./logo.png), is a relative asset path and will be resolved by Webpack as a module dependency.

These assets may be inlined/copied/renamed during build.

Recommended :: put each component in its own directory with its static assets right next to it and continue to use relative path.

Relative URLs
e.g. ./assets/logo.png will be interpreted as a module dependency. They will be replaced with an auto-generated URL based on your Webpack output configuration.
Non-prefixed URLs
e.g. assets/logo.png will be treated the same as the relative URLs and translated into ./assets/logo.png.
URLs prefixed with ~
are treated as a module request, similar to require('some-module/image.png'). You need to use this prefix if you want to leverage Webpack's module resolving configurations. For example if you have a resolve alias for assets, you need to use <img src="~assets/logo.png"> to ensure that alias is respected.
Root-relative URLs
e.g. /assets/logo.png are not processed at all.

Files in ./src/assets/ or any webpack processed assets may be copied to \((build.assetsRoot)\)(build.assetsSubDirectory)

If assets are inserted via JavaScript like computed property, you need to do it in this way 

computed: {
  background () {
    return require('./bgs/' + this.id + '.jpg')
  }
}

Limitation :: all images under ./bgs/ will be included in the final build.

Files in ./static/ will be directly copied to \((build.assetsPublicPath)\)(build.assetsSubDirectory). Refer to vue:webpack:config:index.js

Files in ./static/ should be referenced as absolute path like static[filename]. If $(build.assetsSubDirectory) is changed, you need to change in the reference as well.

9.1.16.7. Integrate with Backend vue:webpack:integrate backend
  1. Proxy requests to a backend

    In config/index.js, edit dev.proxyTable option Proxy the request /api/posts/1 to http://jsonplaceholder.typicode.com/posts/1. 

    module.exports = {
  // ...
  dev: {
    proxyTable: {
      // proxy all requests starting with /api to jsonplaceholder
      '/api': {
        target: 'http://jsonplaceholder.typicode.com',
        changeOrigin: true,
        pathRewrite: {
          '^/api': ''
        }
      }
    }
  }
}

    It uses http-proxy-middleware 

    proxyTable: {
  '**': {
    target: 'http://jsonplaceholder.typicode.com',
    filter: function (pathname, req) {
      return pathname.match('^/api') && req.method === 'GET'
    }
  }
}
  2. Backend path

    Default ./config/index.js vue:webpack:config:index.js 

    var path = require('path')

module.exports = {
  build: {
    index: path.resolve(__dirname, 'dist/index.html'),
    // has to be absolute
    // e.g. path.resolve(__dirname, 'resources/views/index.blade.php'),

    assetsRoot: path.resolve(__dirname, 'dist'), // final after build is /dist/static
    // path.resolve(__dirname, 'public/');

    assetsSubDirectory: 'static', // changing this also means absolute paths for real static assets need to be changed 
    assetsPublicPath: '/',
    productionSourceMap: true
  },
  dev: {
    port: 8080,
    proxyTable: {}
  }
}
9.1.16.8. .vue file vue-loader

.vue file name Add attribute lang to chnage language :: <style lang="sass">

<template> :: default html, 0 or 1 template block. <script> :: default js (ES2015 if babel-loader or buble-loader exists). - or 1 script block. require() can be used. With ES2015, import and export can also be used. The result is to export a Vue.js component option object. An extended constructor created by Vue.extend() can also be exported.

<style> :: default css.

  1. scoped css vue:loader:scoped css

    Scoped to current component. 

    <style scoped>
.example {
  color: red;
}
</style>

<template>
  <div class="example">hi</div>
</template>

    Parent component's styles will not leak into child components. However, a child component's root node will be affected by both the parent's scoped CSS and the child's scoped CSS. So that the parent can style the child root element for layout purpose.

    Deep selector 

    <style scoped>
.a >>> .b { /* ... */ }
</style>

// will be compiled into
.a[data-v-f3f3eg9] .b { /* ... */ }

// may use /deep/ combinator instead for >>> in other pre-processors, like SASS
  2. CSS Modules

    vue:loader:css modules https://vue-loader.vuejs.org/en/features/css-modules.html

9.1.17. Ajax, axios

https://github.com/axios/axios https://rubyplus.com/articles/5051-Using-vue-cli-and-axios-in-Vue-js-2 

npm install axios --save

// just import it and use it wherever you want
<script>
  import axios from 'axios'

  export default {
    name: 'app',
    data: function () {
      return {
        articles: []
      }
    },
    methods: {
      fetchArticles: function () {
        axios.get('http://localhost:3000/articles').then((response) => {
          this.articles = response.data
        }, (error) => {
          console.log(error)
        })
      }
    },
    mounted: function () {
      this.fetchArticles()
    }
  }
</script>

9.2. jQuery

9.2.1. Version, Closure, document ready

jQuery.fn.jquery
  • Closure
(function($){
  // can do something like 
  $.fn.function_name = function(x){};
})(jQuery);
  • Document ready shorthand
$(function(){

});

<script src="other_lib.js"></script>
<script src="jquery.js"></script>
<script>
  $.noConflict()
  jQuery(document).ready(function ($) {
    // Code that uses jQuery's $ can follow here.
  })
  // Code that uses other library's $ can follow here.
</script>

<script>
  jQuery.noConflict();
  (function ($) {

    $(function () {
  // More code using $ as alias to jQuery
    })
  })(jQuery)

  // Other code using $ as an alias to the other library
</script>
9.2.1.1. jQuery.noConflict

Return control of $ back to the other library with a call to $.noConflict(). Old references of $ are saved during jQuery initialization; noConflict() simply restores them. 

var j = jQuery.noConflict(); // different alias

// Do something with jQuery
j( "div p" ).hide();

// Do something with another library's $()
$( "content" ).style.display = "none";

Completely move jQuery to a new namespace in another object. 

var dom = {};
dom.query = jQuery.noConflict( true );
// If for some reason two versions of jQuery are loaded (which is not recommended), calling $.noConflict( true ) from the second version will return the globally scoped jQuery variables to those of the first version.

// Do something with the new jQuery
dom.query( "div p" ).hide();

// Do something with another library's $()
$( "content" ).style.display = "none";

// Do something with another version of jQuery
jQuery( "div > p" ).hide();

9.2.2. Ajax

  • $.get(url[, data] [, success], [, dataType] )
  • $.post(url[, data] [, success], [, dataType] )
$.ajax({
 method: 'POST', // Default: GET
// Prior to 1.9.0, use type instead of method
 url: "some.php", // Default current page
 data: { name: "a", location: "Boston" } // Data send to server. Converted as URL parameters
// if data is not key/value pairs, change processData to false
 processData: true, 
// If data is not key/value pairs (object), say it's a DOMDocument, change it to false

 contentType: 'applicaton/x-www-form-urlencoded;charset=UTF-8', // Default. MIME type

 dataType: "json", // Default is not setting anything. MIME type of response
// > v1.5, this option can convert response to the MIME types you want
// "json xml" :: first it will try to convert to json. if failed, convert ot xml
// "script" :: the response is javascript, and it will be run afer it's received.

// cross domain should be automatically handled
// crossDomain: default false for same-domain requests, true for cross-domain requests

 context: document.body, // provide context for callbacks. If not set, this referrence
// in callbacks is the Ajax request settings

 cache: true, // You can only change cache for HEAD and GET requests.

 headers: {}, // Default. Add (or add to overwrite) request header

// Callback options
 beforeSend: function(jqXHR, settings) {
  // Modify jqXHR object before it is sent.
  // e.g. request header
  // For receiving binary string (image or any binary data)
  // jqXHR.overrideMimeType("text/plain; charset=x-user-defined");
 },
 error: function(jqXHR, textStatus, errorThrown) {
  // if request fails.
 },
 dataFilter: function( data, type) {
  // raw response data
  // dataType option
  // You should sanitize the data here so to match the dataType option
 },
 success: function( data, textStatus, jqXHR ) {
 }, 
 complete: function( jqXHR, textStatus) {}

})
/* Promise callbacks: .done, .fail, .always. In those callbacks, 
`this` reference is the `context` in the request. If `context` was not set, `this` 
will be the Ajax setting.
You can add multiple promise callbacks.
*/
.done(function(data, textStatus, jqXHRmsg) {
// .success() will be deprecated in v3.0


})
.fail(function(jqXHR, textStatus, errorThrown) {
// .error() will be deprecated in v3.0
})
.always(function(data|jqXHR, textStatus, jqXHR|errorThrown) {
// .complete() will be dprecated in v3.0

});

9.2.3. jQuery Selectors, Loop through found elements

You can use all CSS Selectors

http://api.jquery.com/category/selectors/ 

$('li').each(function(i) {
 // i = 0, 1, etc.
 $(this).addClass("foo");
});

// multiple attributes
var langEn = $('head link[rel=alternate][hreflang=en]').attr('href');

First matched element :: var iframeId = $("#div-iframe").find('iframe').get(0).id;

9.2.4. iFrame refer to parent

var parentVideoDiv = parent.jQuery('#videoplayer').parent('.the-content');

9.2.5. Find elements

var $querySelection = '...';
if ($querySelect.length > 0) {
  // not empty
  // if ($querySelect.length) {}
}
9.2.5.1. Get parent element, sibling, closet
var parentDiv = jQuery('#videoplayer').parent('.content');
var siblingDiv = parentDiv.prev(); // preceding sibling
var siblingDiv = parentDiv.next(); // next sibling

.closest starts from the current element, travels up the DOM tree until it finds a match for the supplied selector, returns zero or one element. 

$( "li.item-a" )
  .closest( "ul" )
  .css( "background-color", "red" );
9.2.5.2. Find inside an element, .find, .first, .last
siblingDiv.find(".panel.panel-default");

// by data attribute
var $all = $('#id').find('[data-name="' + js_var + '"]');
$first = $all.first();

9.2.6. Pass $(this) to javascript function

  • $(this) is jQuery object, but this is a normal this
function doFunc(obj) {
  obj.className = 'hello'; // className is a normal javascript function
}
$('li').each(function() {
  doFunc(this);
});

// or
function handler(event) {
  var target = $(event.target);
  if (target.is("li")) {
    target.children().toggle();
  }
}
$("ul").click(handler).find("ul").hide();

9.2.7. General click to element

Any elements with attribute data-target="goto1" with id #goto1 

LiliJs.clickToGo = function(event) {
    var $target = $(event.target);
    var iter = 0;
    var itermax = 10;
    var goto = '';
    while (iter < itermax ) {
  if ($target.data('target')) {
    goto =$target.data('target');
    break;
  }
  else {
    $target = $target.parent();
    iter++;
    console.log('+1');
  }
    }
    if (goto) {
  $("#"+goto).get(0).scrollIntoView();
    }
}

$(function() {
  $('.click-1, .click-2').click(LiliJs.clickToGo);
});

9.2.8. Pointer Events

https://mobiforge.com/design-development/html5-pointer-events-api-combining-touch-mouse-and-pen

mousedown touchstart pointerdown
mouseenter   pointerenter
mouseleave   pointerleave
mousemove touchmove pointermove
mouseout   pointerout
mouseover   pointerover
mouseup touchend pointerup
     

pointerover :: pointer moves over an element (enters its hit test boundaries) pointerenter :: pointer moves over an element or one of its descendants. Differs to pointerover in that it doesn’t bubble

pointerdown :: active buttons state is entered: for touch and stylus, this is when contact is made with screen; for mouse, when a button is pressed pointermove :: pointer changes coordinates, or when pressure, tilt, or button changes fire no other event pointerup :: active buttons state is left: i.e. stylus or finger leaves the screen, or mouse button released

pointercancel :: pointer is determined to have ended, e.g. in case of orientation change, accidental input e.g. palm rejection, or too many pointers

pointerout :: pointer moves out of an element (leaves its hit test boundaries). Also fired after pointerup event for no-hover supported devices, and after pointercancel event pointerleave :: pointer moves out of an element and its descendants

gotpointercapture :: when an element becomes target of pointer lostpointercapture :: when element loses pointer capture 

$('li').on( events [, selector ] [, data ], handler )

There's no .on('hover'). Hover is equal to .on('mouseenter mouseleave')

CSS :hover vs :focus Hover is 'true' when the mouse pointer is over an element. Focus is true if the cursor is in that element. It's possible for hover to be false and focus true (e.g click in a text field then move the mouse away)

9.2.9. Replace class

parentDiv.removeClass('col-md-9').addClass('col-md-12');

Or remove a class if it exists or add a class if it doesn't 

$('#id').toggleClass('col-md-9 col-md-12');

Remove class with wild card 

$("#hello").removeClass (function (index, className) {
    return (className.match(/(^|\s)color-\S+/g) || []).join(' ');
});

9.2.10. Move element

/* Before
.siblingDiv
.parentDiv
*/

siblingDiv.insertAfter(parentDiv);

/* After
.parentDiv
.siblingDiv
*/

9.2.11. Add Style

var selector = jQuery('#id');
selector.attr('style', function(i,s) { 
  var a = ' visible: visible !important;
  if (typeof s !== 'undefined') {
    return s + a;
  }
  else {
    return a;
  }
});

9.2.12. Form

9.2.12.1. serialize

Checkboxes and radio buttons are included only if they're checked. All seriailized form field elements must have name attribute. 

$("#form_id").on("submit", function(e) {
 event.preventDefault();
 console.log($(this).serialize());
});
9.2.12.2. Disabled Field

Disabled fields are not editable, selectable, copyable and they are not submitted. In order to select and copy value of a disabled field, you need to enable it when mouseover and disable it when mouseout.

Can't register events directly on disabled fields. Events can be registered on its parent 

$('.parent_container')
  .mouseover(function() { $(this).find('textarea').prop('disabled',false)})
  .mouseout(function() { $(this).find('textarea').prop('disabled',true)});

9.2.13. Animate

9.2.13.1. Scroll
var offset = $("[ng-controller=aController]").offset();

offset.top -= 100; // offset.left is also available

$("html,body").animate({ 
 scrollTop: offset.top,
 // scrollLeft: offset.left // if it doesn't change, it won't have to be set
});

9.2.14. Viewport Change Event

js:viewport size \((window).resize(function() { consoel.log(\)(window).width()); // integer without unit });

9.2.15. jQuery UI: Dialog jqueryui:dialog

<div id="interstitial">
 <!-- script tag inside the container will be run every time the dialog is opend! 
      Consider remove script tag before .dialog and reattach javascript later.
      Content inside the container will be removed and then added to div.ui-dialog at the bottom of DOM.
 -->
</div>

jQuery(function(){
  jQuery("#interstitial").dialog({
    width: 'auto', /* no scrollbar, auto fit content */
    height:'auto',
    resizable: false,
    draggable: false, /* movable */
    closeOnEscape: true,

    /* Add extra buttons below the content
    buttons: [{
      text: 'ok',
      icons: {primary: "ui-icon-closethick"},
      click: function() {jQuery(this).dialog("close");}
    }],
    */

    modal: true, /* add background to isolate the modal box */
    create: function (event, ui) {
      jQuery("#ui-dialog-title-dialog").hide();
      /* custom background/overlay */
      jQuery(".ui-widget-content").css({"background-color":"transparent","background":"transparent","border":"none"});
      /* title bar is transparent */
      jQuery(".ui-dialog-titlebar").css({"background-color":"transparent","background":"transparent","border":"none"});

      jQuery(".ui-dialog-titlebar").removeClass('ui-widget-header');
      // Change the default title & close button to a custom link
      //jQuery(".ui-dialog-titlebar").html('<a href="#" role="button"><span class="myCloseIcon" style="color:white;">X close</span></a>')

      // You can still load/modify the content here and the width/height will still be auto fit
    },
    open: function() {
      /* custom background */
      jQuery('.ui-widget-overlay').addClass('custom-overlay');

      /* Customize the default close button: only X displays */
      jQuery(this).closest(".ui-dialog")
        .find(".ui-dialog-titlebar-close")
        .removeClass("ui-dialog-titlebar-close")
        .html("<span class='ui-button-icon-primary ui-icon ui-icon-closethick'></span>");
      jQuery(".ui-dialog-titlebar button").css({"border":"none"});

      /* Close dialog when click outside */
      jQuery('.ui-widget-overlay').bind('click',function(){
        jQuery("#interstitial").dialog('close');
      })
    },
    close: function() {
      /* custom background/overlay */
      /*
      * .ui-widget-overlay.custom-overlay
       {
       background-color: black;
       background-image: none;
       opacity: 0.9;
       }
      * */
      jQuery('.ui-widget-overlay').removeClass('custom-overlay');
    }
  });

  /* bind events for custom buttons created
  jQuery("span.myCloseIcon").click(function() {
    jQuery("#interstitial").dialog( "close" );
  });
  */
});

9.2.16. jQuery UI: Sortable

$(function() {
    $("#slideshow-pictures").sortable({
      items:'.dz-preview',
      cursor: 'move',
      opacity: 0.5,
      containment: '#slideshow-pictures',
      distance: 20,
      tolerance: 'pointer'
    });
  });

9.3. TweenLite, TweenMax, Greensock

TweenMax has other common plugins included e.g. CSSPlugin where you can ease CSS property changes.

CSSPlugin
change individual DOM element's style attribute
(no term)
RoundPropsPlugin
(no term)
BezierPlugin
(no term)
AttrPlugin
(no term)
DirectionalRotationPlugin
(no term)
EasePack
(no term)
TimelineLite
(no term)
TimelineMax

Extra plugins which are separate js files loaded after TweenMax https://greensock.com/tweenmax:

CSSRulePlugin
change style sheet rules e.g. for class .myClass and other pseudo elements :after :before which are impossible to reference directly in JavaScript
(no term)
modifiers
(no term)
text
(no term)
scrollTo
(no term)
pixi
(no term)
easel
Draggable
Spin, Scrll
(no term)
jquery.gsap.js

Using TweenLite only and you'll have to manually include these plugins.

All versions on GitHub

9.3.1. TweenLite.to() TweenLite.from() TweenLite.fromTo()

https://greensock.com/docs/TweenLite/static.to 

TweenLite.to( [target object], [duration in seconds], [destination values] );

TweenLite.to(active_slide, 0.5, {marginLeft:'-50%',ease:Power3.easeOut});

TweenLite.to("#myID", 1, {left:"100px"});
// TweenLite will use selector engine e.g. jQuery if present or document.querySelectorAll() or lastly document.getElementById()

TweenLite.to(".myClass", 2, {boxShadow:"0px 0px 20px red", color:"#FC0"});

// multiple objects
TweenLite.to([obj1, obj2, obj3], 1, {opacity:0.5, rotation:45});

TweenLite.from() define the starting values instead of the ending values. TweenLite.fromTo() :: TweenLite.fromTo( target:Object, duration:Number, fromVars:Object, toVars:Object ) 

var tween = new TweenLite(element, 2, {width:200, height:150});
// or
var tween = TweenLite.to(element, 2, {width:200, height:150});
9.3.1.1. dest:css, relative value

It's important that you remove all dashes (-) from the css property names and use camelCase. 

TweenLite.to(logo, 2, {left:"542px",
                       backgroundColor:"black",
                       borderBottomColor:"#90e500",
                       color:"white"});

Adding a += or -= prefix tells GSAP to treat a value as relative to whatever it is when the tween renders the first time. For example, if "left" is currently 50px and you tween to "+=100px", it will end up at 150px. Don't forget to put quotes around the value. 

TweenLite.to(logo, 0.5, {left:"+=100px"});
9.3.1.2. dest:ease, easeIn (begin), easeOut (the end), easeInOut (both)

https://greensock.com/docs/Easing

Default is Quad.easeOut

  • Linear
  • 0 is linear
  • Quad
  • Cubic
  • Quart
  • Quint
  • Strong

Extras

  • Elastic Back Bounce SlowMo SteppedEase Rough Circ Expo Sine

Variants

  • .easeIn .easeOut .easeInOut .easeNone

For linear animation, use Linear.easeNone

Sample 

TweenLite.to(logo, 3, {left:"440px", ease:Bounce.easeOut});

ease:Bounce.easeOut
ease:"Strong.easeOut"
ease:"easeOutStrong"
9.3.1.3. dest:delay in seconds, dest:paused bool, dest:immediateRender
paused
If true, the animation will pause immediately upon creation
immediateRender
9.3.1.4. dest:onComplete onCompleteParams
TweenLite.to(element, 1, {left:100, onComplete:myFunction});
TweenLite.to(element, 1, {left:100, onComplete:myFunction, onCompleteParams:[element, "param2"]});

// pass the tween instance use {self}
TweenLite.to(element, 1, {left:100, onComplete:myFunction, onCompleteParams:["{self}", "param2"]});

9.3.2. Use Case

9.3.2.1. Slideshow
<section id="slider-wrap" class="home-slide inner">

  <div id="slides-frame">

    <div id="active-slide" class="slide-img"
         style="background:transparent url('1.jpg') no-repeat center 15%;background-size:cover;-webkit-transform: translateZ(0);">
      <div class="background-screen">
        <div class="page-wrap">
          <div class="slide-headline">
            <h2>Slide #1 Title</h2>
            <p class="quote">Slide #1 Paragraph</p>
            <p class="quote-name">Slide #1 Paragraph</p>
            <a href="#">Slide #1 Button</a>
          </div>
        </div>
      </div>
    </div>

    <div id="next-slide" class="slide-img"
         style="background:transparent url('2.jpg') no-repeat center 25%;background-size:cover;-webkit-transform: translateZ(0);">
      <div class="background-screen">
        <div class="page-wrap">
          <div class="slide-headline">
            <h2>Slide #2 Title</h2>
            <p class="quote">Slide #2 Paragraph</p>
            <p class="quote-name">Slide #2 Paragraph</p>
            <a href="#">Slide #2 Button</a>
          </div>
        </div>
      </div>
    </div> <!-- #next-slide -->

  </div> <!-- #slides-frame -->
</section>

<!-- @@ Slideshow Data: -->
<div id="slideshow-data" style="display:none!important;">
  <div class="slideshow-data-item"
       style="background:transparent url('1.jpg') no-repeat center 15%;background-size:cover;-webkit-transform: translateZ(0);">
    <div class="background-screen">
      <div class="page-wrap">
        <div class="slide-headline">
          <h2>Slide #1 Title</h2>
          <p class="quote">Slide #1 Paragraph</p>
          <p class="quote-name">Slide #1 Paragraph</p>
          <a href="#">Slide #1 Button</a>
        </div>
      </div>
    </div>
  </div>

  <div class="slideshow-data-item"
       style="background:transparent url('2.jpg') no-repeat center 25%;background-size:cover;-webkit-transform: translateZ(0);">
    <div class="background-screen">
      <div class="page-wrap">
        <div class="slide-headline">
          <h2>Slide #2 Title</h2>
          <p class="quote">Slide #2 Paragraph</p>
          <p class="quote-name">Slide #2 Paragraph</p>
          <a href="#">Slide #2 Button</a>
        </div>
      </div>
    </div>
  </div>

  <div class="slideshow-data-item"
       style="background:transparent url('3.jpg') no-repeat center left;background-size:cover;-webkit-transform: translateZ(0);">
    <div class="background-screen">
      <div class="page-wrap">
        <div class="slide-headline">
          <h2>Slide #3 Title</h2>
          <p class="quote">Slide #3 Paragraph</p>
          <p class="quote-name">Slide #3 Paragraph</p>
          <a href="#">Slide #3 Button</a>
        </div>
      </div>
    </div>
  </div>

</div> <!-- #slideshow-data -->

.page {
  max-width:1200px;
  width:90%;
  margin:0 auto;
}
.home-slide.inner {
  height: 550px; /* media query change height */
  overflow: hidden;
}
#slides-frame {
  width:200%!important;
  position:relative;
  height:550px;
}
#active-slide, #next-slide {
  position:relative;
  display:inline-block;
  width:50%;
  height:100%;
  vertical-align:top;
}
#slides-frame .background-screen {
  position:relative;
  height:100%;
  width:100%;
  overflow-y:hidden;
  background:rgba(0,0,0,0.35); /* darken bg image */
}
.slide-wrap {
  /* Text content width */
  width:1200px; /* media query change width */
  /* width:100%; */
  height:100%;
  overflow:visible;
  margin:0 auto;
}
.slide-headline {
  position:absolute;
  left:0;
  bottom:0;
  margin-bottom:155px; /* media query change margin-bottom */
}

window.onload = init();
function init() {
  slideshow_checker();
}
function slideshow_checker() {
  var slideshow_data = document.getElementById('slider-wrap');
  if (slideshow_data != null) {
    slideshow(0, 'time', 7000);
  }
}

var timer;
function slideshow(id, type, time_length) {

  var active_slide = document.getElementById('active-slide'),
    active_slide_style = active_slide.getAttribute('style');
  var next_slide = document.getElementById('next-slide');


  /* -- COUNT SLIDE NUM -- */
  var slideshow_data = document.getElementById('slideshow-data');
  var slides = slideshow_data.getElementsByClassName('slideshow-data-item');
  var slide_count = slides.length;

  var nxt_id, nxt_nxt_id, prev_id;
  nxt_id = (id + 1);
  prev_id = (id - 1);

  if (nxt_id == slide_count) {
    nxt_id = 0;
  }
  if ((id == 0)) {
    prev_id = (slide_count - 1);
  }

  //  arr_left.setAttribute('onclick','javascript:get_slide('+prev_id+')');
  //    arr_right.setAttribute('onclick','javascript:get_slide('+nxt_id+')');

  if (type == 'click') {
    var next_slide_data = slides[id].getAttribute('style');
    var next_slide_html = slides[id].innerHTML;
    next_slide.innerHTML = next_slide_html;
    next_slide.setAttribute('style', next_slide_data);

  }
  else {
    var next_slide_data = slides[nxt_id].getAttribute('style');
    var next_slide_html = slides[nxt_id].innerHTML;
    next_slide.innerHTML = next_slide_html;
    next_slide.setAttribute('style', next_slide_data);
  }

  timer = setTimeout(function () {
    TweenLite.to(active_slide, 0.5, {marginLeft: '-50%', ease: Power3.easeOut});
    setTimeout(function () {

      active_slide.setAttribute('style', next_slide.getAttribute('style'));

      TweenLite.to(active_slide, 0, {marginLeft: '0%'});

      active_slide.innerHTML = next_slide_html;

      /* -- RUN LOOP() --- */
      setTimeout(function () {
        if (type == 'time') {
          slideshow(nxt_id, 'time', 7000);
        }
        else {
          slideshow(nxt_id, 'time', time_length);
        }
      }, 0);

    }, 1000);
  }, time_length);

}

9.4. Select2 js:lib:select2

https://select2.org/ depends on jQuery

9.4.1. Basic

<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script>
<link href="https://cdnjs.cloudflare.com/ajax/libs/select2/4.0.6-rc.0/css/select2.min.css" rel="stylesheet" />
<script src="https://cdnjs.cloudflare.com/ajax/libs/select2/4.0.6-rc.0/js/select2.min.js"></script>

<select class="js-example-basic-single" name="state">
  <option value="AL">Alabama</option>
    ...
  <option value="WY">Wyoming</option>
</select>

<script>
$(document).ready(function() {
    $(".js-example-basic-single").select2();
});
</script>

9.4.2. Internal Option data, <optgroup>

Select2 converts each <option> in this JSON 

{
  "id": "value attribute" || "option text",
  "text": "label attribute" || "option text",
  "element": HTMLOptionElement
}

//"<optgroup>"

{
  "text": "label attribute",
  "children": [ option data object, ... ],
  "element": HTMLOptGroupElement
}

<select>
  <optgroup label="Group Name">
    <option>Nested option</option>
  </optgroup>
</select>

9.4.3. config:ajax

Supply JSON to Select2 

{
  "results": [
    {
      "id": 1,
      "text": "Option 1"
    },
    {
      "id": 2,
      "text": "Option 2",
      "selected": true
    },
    {
      "id": 3,
      "text": "Option 3",
      "disabled": true
    }
  ]
}

ajax is triggered every time the user types in the search box.

By default, these parameters are sent term :: search term q :: same as term _type :: a request type page ::

e.g. https://api.github.com/search/repositories?term=sel&_type=query&q=sel 

$('#mySelect2').select2({
  ajax: {
    url: 'https://api.github.com/orgs/select2/repos',
    data: function (params) {
      var query = {
        search: params.term,
        type: 'public'
      }

      // Query parameters will be ?search=[term]&type=public
      return query;
    }
  }
});

$('#mySelect2').select2({
  ajax: {
    url: '/example/api',
    processResults: function (data) {
      // Tranforms the top-level key of the response object from 'items' to 'results'
      return {
        results: data.items
      };
    }
  }
});

Pagination 

$('#mySelect2').select2({
  ajax: {
    url: 'https://api.github.com/search/repositories',
    data: function (params) {
      var query = {
        search: params.term,
        page: params.page || 1
      }

      // Query parameters will be ?search=[term]&page=[page]
      return query;
    }
  }
});

If the response has pagination.more prop 

{
  "results": [
    {
      "id": 1,
      "text": "Option 1"
    },
    {
      "id": 2,
      "text": "Option 2"
    }
  ],
  "pagination": {
    "more": true
  }
}

If not, then do it manually. response provides count_filtered that shows the total number. 

processResults: function (data, params) {
    params.page = params.page || 1;

    return {
        results: data.results,
        pagination: {
            more: (params.page * 10) < data.count_filtered
        }
    };
}

9.4.4. config:data

Load local Javascript array to Select2 

var data = [
    {
        id: 0,
        text: 'enhancement'
    },
    {
        id: 1,
        text: 'bug'
    },
    {
        id: 2,
        text: 'duplicate'
    },
    {
        id: 3,
        text: 'invalid'
    },
    {
        id: 4,
        text: 'wontfix'
    }
];

$(".js-example-data-array").select2({
  data: data
})

$(".js-example-data-array-selected").select2({
  data: data
})

9.4.5. config:templateResult, Template

function formatState (state) {
  if (!state.id) {
    return state.text;
  }
  var baseUrl = "/user/pages/images/flags";
  var $state = $(
    '<span><img src="' + baseUrl + '/' + state.element.value.toLowerCase() + '.png" class="img-flag" /> ' + state.text + '</span>'
  );
  return $state;
};

$(".js-example-templating").select2({
  templateResult: formatState
});

9.4.6. config:templateSelection, Change display of selected value

function formatState (state) {
  if (!state.id) {
    return state.text;
  }
  var baseUrl = "/user/pages/images/flags";
  var $state = $(
    '<span><img src="' + baseUrl + '/' + state.element.value.toLowerCase() + '.png" class="img-flag" /> ' + state.text + '</span>'
  );
  return $state;
};

$(".js-example-templating").select2({
  templateSelection: formatState
});

9.4.7. config:maximumSelectionLength

$(".js-example-basic-multiple-limit").select2({
  maximumSelectionLength: 2
});

9.4.8. config:placeholder

First option has to be empty without any selected option 

<select class="js-example-placeholder-single js-states form-control">
  <option></option>
</select>

$(".js-example-placeholder-single").select2({
    placeholder: "Select a state",
    allowClear: true
});

placeholder can be an existing <option> 

$('select').select2({
  placeholder: {
    id: '-1', // the value of the option
    text: 'Select an option'
  }
});

9.4.9. config:allowClear

Add a x for clearing 

$('select').select2({
  placeholder: 'This is my placeholder',
  allowClear: true
});

9.4.10. config:width

Default is 'resolve' which is to take the style attribute of <select>, if not, fall back to computed element width ('element').

To make sure the field always takes 100% width of parent, use this 

'width: '100%'

9.4.11. config:tags

User can type a new option and then select it 

<select class="form-control">
  <option selected="selected">orange</option>
  <option>white</option>
  <option>purple</option>
</select>

$(".js-example-tags").select2({
  tags: true
});

// multiple
<select class="form-control" multiple="multiple">
  <option selected="selected">orange</option>
  <option>white</option>
  <option selected="selected">purple</option>
</select>

9.4.12. config:tokenSeparators

User can type a space or a comma to add existing or type new option. 

$(".js-example-tokenizer").select2({
    tags: true,
    tokenSeparators: [',', ' ']
})

9.4.13. config:createTag

After a new option is created by typing in config:tags, add properties to this new option Reject creating a new option by returning null 

$('select').select2({
  createTag: function (params) {
    var term = $.trim(params.term);

    if (term === '') {
      return null;
    }

    return {
      id: term,
      text: term,
      newTag: true // add additional parameters
    }
  }
});

9.4.14. config:insertTag

After a new option is created by typing in config:tags, do something about it 

$('select').select2({
  insertTag: function (data, tag) {
    // Insert the tag at the end of the results
    data.push(tag);
  }
});

9.4.15. config:matcher, config:minimumInputLength, config:minimumResultsForSearch, Search

For single select, a search box is added to search each option. This search can be customized.

It only works for locally supplied data. 

function matchCustom(params, data) {
    // If there are no search terms, return all of the data
    if ($.trim(params.term) === '') {
      return data;
    }

    // Do not display the item if there is no 'text' property
    if (typeof data.text === 'undefined') {
      return null;
    }

    // `params.term` should be the term that is used for searching
    // `data.text` is the text that is displayed for the data object
    if (data.text.indexOf(params.term) > -1) {
      var modifiedData = $.extend({}, data, true);
      modifiedData.text += ' (matched)';

      // You can return modified objects from here
      // This includes matching the `children` how you want in nested data sets
      return modifiedData;
    }

    // Return `null` if the term should not be displayed
    return null;
}

$(".js-example-matcher").select2({
  matcher: matchCustom,
  minimumInputLength: 3, // only start searching when the user has input 3 or more characters
  minimumResultsForSearch: 20 // at least 20 results must be displayed. -1 or Infinity to permanently hide the search box.
  // minimumResultsForSearch: Infinity,
});

9.4.16. Methods, add, select, clear

add 

// after $.append, <select> will have the new option

<select id="mySelect2">
var data = {
    id: 1,
    text: 'Barn owl'
};

var newOption = new Option(data.text, data.id, false, false);
$('#mySelect2').append(newOption).trigger('change');

// create if not exists

// Set the value, creating a new option if necessary
if ($('#mySelect2').find("option[value='" + data.id + "']").length) {
    $('#mySelect2').val(data.id).trigger('change');
} else { 
    // Create a DOM Option and pre-select by default
    var newOption = new Option(data.text, data.id, true, true);
    // Append it to the select
    $('#mySelect2').append(newOption).trigger('change');
}

select 

$('#mySelect2').val('1'); // Select the option with a value of '1'
$('#mySelect2').trigger('change'); // Notify any JS components that the value changed

$('#mySelect2').val(['1', '2']);
$('#mySelect2').trigger('change'); // Notify any JS components that the value changed

clear 

$('#mySelect2').val(null).trigger('change');

9.4.17. Method, get

// returns a array of objects
$('#mySelect2').select2('data');
$('#mySelect2').find(':selected');

// get value
$('#mySelect2').val();

// pure javascript will not work
// document.querySelector('#mySelect2').value;

$('#mySelect2').select2({
  // ...
  templateSelection: function (data, container) {
    // Add custom attributes to the <option> tag for the selected option
    $(data.element).attr('data-custom-attribute', data.customValue);
    return data.text;
  }
});

// Retrieve custom attribute value of the first selected element
$('#mySelect2').find(':selected').data('custom-attribute');

9.4.18. Method, open|close dropdown, if initialized, destroy

$('#mySelect2').select2('open');

$('#mySelect2').select2('close');

if ($('#mySelect2').hasClass("select2-hidden-accessible")) {
    // Select2 has been initialized
}

$('#mySelect2').select2('destroy');

Custom events need to be unbinded after destroy 

// Set up a Select2 control
$('#example').select2();

// Bind an event
$('#example').on('select2:select', function (e) { 
    console.log('select event');
});

// Destroy Select2
$('#example').select2('destroy');

// Unbind the event
$('#example').off('select2:select');

9.4.19. Example: Destroy and Initiate

<button class="js-programmatic-destroy button">Destroy</button>
<button class="js-programmatic-init button">Re-initialize</button>

<script>
$(".js-programmatic-destroy").on("click", function () {
    $example.select2("destroy");
});

$(".js-programmatic-init").on("click", function () {
    $example.select2();
});
</script>

9.4.20. Add down arrow to multiple <select>

By default, multiple <select> doesn't have a down arrow. The solution is to copy .select2-selection--single to .select2-selection--multiple 

.select2-container--default .select2-selection--multiple .select2-selection__rendered{
  padding-left:8px;
  padding-right:20px;
}
.select2-container--default .select2-selection--multiple .select2-selection__clear{
  font-size:13px;
  color:#666;
  margin:0;
  line-height:26px;
}
.select2-selection--multiple:after{
  content:"";
  position:absolute;
  right:7px;
  top:12px;
  width:0;
  height:0;
  border-left: 4px solid transparent;
  border-right: 4px solid transparent;
  border-top: 5px solid #888;
}

9.4.21. Events

change :: general Javascript select change change.select2 :: only for Select2 <select> change select2:selecting :: Triggered before a result is selected. This event can be prevented. select2:select :: Triggered whenever a result is selected. select2:selecting is fired before this and can be prevented. 

$('#mySelect2').on('select2:select', function (e) {
  var data = e.params.data;
  console.log(data);
});

Trigger an event 

var data = {
  "id": 1,
  "text": "Tyto alba",
  "genus": "Tyto",
  "species": "alba"
};

$('#mySelect2').trigger({
    type: 'select2:select',
    params: {
        data: data
    }
});

$('#mySelect2').trigger('change.select2'); // Notify only Select2 of changes

9.5. Owl Carousel

https://github.com/OwlCarousel2/OwlCarousel2

Bootstrap's carousel can only show one item like a slideshow. Owl can have multiple items.

Example: Owl Carousel + Bootstrap Modal + Bootstrap Carousel

9.6. Fine Uploader

https://github.com/FineUploader/fine-uploader No dependency

9.7. Dropzone

https://github.com/enyo/dropzone/

No dependency

9.7.1. Basics

Download the dist folder 

<script src="/dist/dropzone.js"></script>
<link rel="stylesheet" href="/dist/dropzone.css">
<!-- /dist/min/* for production -->

Dropzone will be available as window.Dropzone and jQuery.fn.Dropzone (if jQuery is loaded)

Create Dropzone without javascript (auto discover), Dropzone.options 

<form action="/file-upload" class="dropzone" id="my-awesome-dropzone">
  <!-- hidden input will be submitted -->
  <!-- <input type="hidden" name="addtionaldata" value="1" /> -->
  <div class="fallback">
    <input name="file" type="file" multiple />
  </div>
</form>

<!-- because it's auto discovered, you may need to setup the config -->
// "myAwesomeDropzone" is the camelized version of the HTML element's ID
Dropzone.options.myAwesomeDropzone = {
  paramName: "file", // The name that will be used to transfer the file
  maxFilesize: 2, // MB
  accept: function(file, done) {
    if (file.name == "justinbieber.jpg") {
      done("Naha, you don't.");
    }
    else { done(); }
  }
};

Create with javascript, must have url option. 

// Prevent Dropzone from auto discovering this element:
Dropzone.options.myAwesomeDropzone = false;
// This is useful when you want to create the
// Dropzone programmatically later

// Disable auto discover for all elements:
Dropzone.autoDiscover = false;

// Dropzone class:
var myDropzone = new Dropzone("div#myId", { url: "/file/post"});

// jQuery
$("div#myId").dropzone({ url: "/file/post" });

The file will be posted in backend as 

<input type="file" name="file" />

To make the whole page droppable 

<!--
Don't forget to give this container the dropzone-previews class 
so the previews are formatted correctly.
 -->
<div id="previews" class="dropzone-previews"></div>

<button id="clickable">Click me to select files</button>

var myDropzone  = new Dropzone(document.body, {
  previewsContainer: ".dropzone-previews", // the element must have
  // You probably don't want the whole body
  // to be clickable to select files
  clickable: false
});

// or
<script>
  new Dropzone(document.body, { // Make the whole body a dropzone
    url: "/upload/url", // Set the url
    previewsContainer: "#previews", // Define the container to display the previews
    clickable: "#clickable" // Define the element that should be used as click trigger to select files.
  });
</script>

9.7.2. option:function:accept(file, done)

Invoke done() without arguments to accept and process the file. Invoke done() with an error message to reject the file and show the message. It won't be invoked if the file is too big or mime type doesn't match. 

accept: function(file, done) {
  if (file.name == "justinbieber.jpg") {
    done("Naha, you don't.");
  }
  else { done(); }
}

9.7.3. option:previewsContainer null|string

HTMLElement or a CSS selector. The element should have the .dropzone-previews or dropzone class.

9.7.4. option:autoProcessQueue

False :: files will be added to the queue but the queue will not be processed automatically. Call myDropzone.processQueue() to process the queue.

9.7.5. option:previewTemplate String, option:dict*

Default template dropzone:default template 

<div class="dz-preview dz-file-preview">
  <div class="dz-details">
    <div class="dz-filename"><span data-dz-name></span></div>
    <div class="dz-size" data-dz-size></div>
    <!-- img's alt and src will change -->
    <img data-dz-thumbnail />
  </div>
  <div class="dz-progress"><span class="dz-upload" data-dz-uploadprogress><!-- style.width from 0% to 100% --></span></div>
  <div class="dz-success-mark"><span>✔</span></div>
  <div class="dz-error-mark"><span>✘</span></div>
  <div class="dz-error-message"><span data-dz-errormessage></span></div>
  <!-- <a class="dz-remove" href="javascript:undefined" data-dz-remove>Remove file</a> -->
</div>
<!-- Another preview -->

The default template might change from version to version. But console.log(file.previewElement) in any event will output.

.dz-preview gets .dz-processing when the file gets processed, .dz-success when it got uploaded and .dz-error

Make sure custom template has these attributes

  • data-dz-name
  • data-dz-size
  • data-dz-thumbnail
  • data-dz-uploadprogress
  • data-dz-errormessage

Add a remove button (data-dz-remove) 

<img src="removebutton.png" alt="Click me to remove the file." data-dz-remove />

9.7.6. option:addRemoveLInks null:true

Wording: dictCancelUpload, dictCancelUploadConfirmation and dictRemoveFile dropzone:default template

9.7.7. option:translation

9.7.7.1. option:dictCancelUpload, dictCancelUploadConfirmation, dictRemoveFile

9.7.8. option:maxFiles int

Dropzone.options.myAwesomeDropzone = {
  maxFiles: 1,
  accept: function(file, done) {
    console.log("uploaded");
    done();
  },
  init: function() {
    this.on("maxfilesexceeded", function(file){
        alert("No more files please!");
      this.removeFile(file);
    });
  }
};

9.7.9. option:maxFilesize int (MB)

event:maxfilesexceeded is called and the whole dropzone element gets the class .dz-max-files-reached

9.7.10. option:event:init, add events

It's the best place to add custom events if options are used to setup the dropzone 

// The recommended way from within the init configuration:
Dropzone.options.myAwesomeDropzone = {
  init: function() {
    this.on("addedfile", function(file) { alert("Added file."); });
  }
};

Use this to attach events if dropzone is defined programmatically 

// This example uses jQuery so it creates the Dropzone, only when the DOM has
// loaded.

// Disabling autoDiscover, otherwise Dropzone will try to attach twice.
Dropzone.autoDiscover = false;
// or disable for specific dropzone:
// Dropzone.options.myDropzone = false;

$(function() {
  // Now that the DOM is fully loaded, create the dropzone, and setup the
  // event listeners
  var myDropzone = new Dropzone("#my-dropzone");
  myDropzone.on("addedfile", function(file) {
    /* Maybe display some more file information on your page */
  });
})

9.7.11. option:clickable null|true|false|an html element|a CSS selector|array of html

All of those elements will trigger an upload when clicked

9.7.12. option:acceptedFiles string of list

comma separated list of mime types of file extensions 

image/*,applicaton/pdf,.psd

9.7.13. option:maxThumbnailFilesize (MB), thumbnailWidth, thumbnailHeight, thumbnailMethod

thumbnailMethod :: 'contain' or 'crop' when thumbnailWidth and thumbnailHeight are both defined!

9.7.14. event:addedfile

// Disabling autoDiscover, otherwise Dropzone will try to attach twice.
Dropzone.autoDiscover = false;
// or disable for specific dropzone:
// Dropzone.options.myDropzone = false;

$(function() {
  // Now that the DOM is fully loaded, create the dropzone, and setup the
  // event listeners
  var myDropzone = new Dropzone("#my-dropzone");
  myDropzone.on("addedfile", function(file) {
    /* Maybe display some more file information on your page */
  });
})

9.7.15. event:removedfile(file)

9.7.16. event:thumbnail(file, dataUrl)

9.7.17. event:processing(file)

when a file gets processed

9.7.18. event:uploadprogress(file, progress, bytesSent)

progress :: int from 0 to 100

9.7.19. event:error(file)

9.7.20. event:success(file, responseText)

Display something after a file is uploaded.

myDropzone.files has been appended. 

Dropzone.options.myDropzone = {
  init: function() {
    this.on("success", function(file, responseText) {
      // Handle the responseText here. For example, add the text to the preview element:
      file.previewTemplate.appendChild(document.createTextNode(responseText));
    });
  }
};

Display thumbnail that is created on server after server processes the uploaded image { "imageUrl": "http://my.image/file.jpg" } 

Dropzone.options.myDropzone = {
  init: function() {
    this.on("success", function(file, serverResponse) {
      // Called after the file successfully uploaded.

      // If the image is already a thumbnail:
      this.emit('thumbnail', file, serverResponse.imageUrl);

      // If it needs resizing:
      this.createThumbnailFromUrl(file, serverResponse.imageUrl);
    });
  }
};

9.7.21. event:complete(file)

9.7.22. event:canceled(file)

9.7.23. event:maxfilesexceeded(file)

9.7.24. event:maxfilesreached(file)

the number of files accepted reaches the maxFiles limit

9.7.26. Show error returned by server

Just return HTTP status code in the range of 400-500. If response Content-Type is text/plain, the text will be returned as error message. If it's application/json, dropzone uses the error property {"error": "File could not be saved."}

9.7.27. Show server response

See event:success(file, responseText)

9.7.28. Show files already stored on server

// Create the mock file:
var mockFile = { name: "Filename", size: 12345 };

// Call the default addedfile event handler
myDropzone.emit("addedfile", mockFile);

// And optionally show the thumbnail of the file:
myDropzone.emit("thumbnail", mockFile, "/image/url");
// Or if the file on your server is not yet in the right
// size, you can let Dropzone download and resize it
// callback and crossOrigin are optional.
myDropzone.createThumbnailFromUrl(file, imageUrl, callback, crossOrigin);

// Make sure that there is no progress bar, etc...
myDropzone.emit("complete", mockFile);

myDropzone.files.push(mockFile);

// If you use the maxFiles option, make sure you adjust it to the
// correct amount:
var existingFileCount = 1; // The number of files already uploaded
myDropzone.options.maxFiles = myDropzone.options.maxFiles - existingFileCount;

9.7.29. Sortable

  $(function() {
    $("#slideshow-pictures").sortable({
      items:'.dz-preview',
      cursor: 'move',
      opacity: 0.5,
      containment: '#slideshow-pictures',
      //distance: 20,
      tolerance: 'pointer',
      update: function (event, ui) {
        var queue = myDropzone.files;
        var newQueue = [];
        $('#slideshow-pictures .dz-preview .dz-filename [data-dz-name]').each(function (count, el) {
          var name = el.innerHTML;
          queue.forEach(function(file) {
            if (file.name === name) {
              newQueue.push(file);
            }
          });
        });
        myDropzone.files = newQueue;
      }
    });
  });

Dropzone.options.slideshowPictures = false;
Dropzone.autoDiscover = false;
var myDropzone = new Dropzone("#slideshow-pictures", {...});

9.8. modernizr

It usually adds a class in <html> when a certain feature is supported 

// <html class="no-cssgradients">
// <html class="cssgradients">

.no-cssgradients .header {
  background: url("images/glossybutton.png");
}
.cssgradients .header {
  background-image: linear-gradient(cornflowerblue, rebeccapurple);
}

Config 

{
  "classPrefix": "foo-",
  "feature-detects": ["dom/hidden"]
}

9.8.1. Feature

Modernizr.feature-name is true or false, true add .classPrefix-feature-name, false add .classPrefix-no-feature-name 

if (Modernizr.mediaqueries) {
  // supported
} else {
  // not-supported
}
9.8.1.1. flexbox, flexboxlegacy, flexboxtweener, flexwrap modernizr:flexbox
.flexbox.no-flexwrap .row {
  display:-webkit-flex-wrap: wrap;

}

Bootstrap 4 fallback flex 

Lili.cssFixFlex = function() {
  if (!Modernizr.testProp('display', 'flex') && !Modernizr.testProp('display', '-ms-flexbox') && Modernizr.testProp('display', '-webkit-box') && Modernizr.testProp('display', '-webkit-flex')) {
    // display:flex and -ms-flexbox are not supported but -webkit-box and -webkit-flex are supported
    // Bootstrap uses display:-webkit-box only
    // add class to html tag
    $('html').addClass('bootstrap-ios-fix-row');
  }
  else if (!Modernizr.testProp('display', 'flex') && !Modernizr.testProp('display', '-ms-flexbox') && Modernizr.testProp('display', '-webkit-box') && !Modernizr.testProp('display', '-webkit-flex')) {
    // flex is completely not supported
    $('html').addClass('bootstrap-fix-row');
  }
}

$(function() {
  Lili.cssFixFlex();
})

.bootstrap-ios-fix-row .row {
  display:-webkit-flex;
  -webkit-flex-wrap:wrap;
}
.bootstrap-fix-row .row {
  display:inline-block;
}
9.8.1.2. touchevents
9.8.1.3. videoautoplay

This feature detection is not reliable..

9.8.1.4. webp modernizr:webp

Modernizr.webp, Modernizr.webp.lossless, Modernizr.webp.alpha and Modernizr.webp.animation

9.8.2. Option

9.8.2.1. .mq(mq)
var query = Modernizr.mq('(min-width: 900px)');
if (query) {
  // the browser window is larger than 900px
}

Modernizr.mq('only all'); // true if MQ are supported, false if not

9.8.3. API

9.8.3.1. Modernizr.on(feature, cb)
Modernizr.on('flash', function( result ) {
  if (result) {
   // the browser has flash
  } else {
    // the browser does not have flash
  }
});
9.8.3.2. .addTest('customFeatureName', cb)
Modernizr.addTest('itsTuesday', function() {
 var d = new Date();
 return d.getDay() === 2;
});

Modernizr.addTest('hasJquery', 'jQuery' in window);

// combine
var detects = {
 'hasjquery': 'jQuery' in window,
 'itstuesday': function() {
   var d = new Date();
   return d.getDay() === 2;
 }
}
Modernizr.addTest(detects);
9.8.3.3. .atRule(prop)
var keyframes = Modernizr.atRule('@keyframes');
if (keyframes) {
   // keyframes are supported
   // could be `@-webkit-keyframes` or `@keyframes`
} else {
  // keyframes === `false`
}
9.8.3.4. ._domPrefixes, ._prefixes

._prefixes return kebab-case properties 

var rule = Modernizr._prefixes.join('transform: rotate(20deg); ');
rule === 'transform: rotate(20deg); webkit-transform: rotate(20deg); moz-transform: rotate(20deg); o-transform: rotate(20deg); ms-transform: rotate(20deg);'

Modernizr._domPrefixes = [ "Moz", "O", "ms", "Webkit" ];

9.8.3.5. .hasEvent(eventName, [element])
9.8.3.6. .prefixedCSS(prop), .prefixed(prop, [obj], [elem])
Modernizr.prefixedCSS('transition') // '-moz-transition' in old Firefox
9.8.3.7. .prefixedCSSValue(prop,value)
9.8.3.8. .testAllProps(prop,[value], [skipValueTest]), .testProp(prop,[value], [useValue])

Whether a given CSS property, in some prefixed form, is supported by the browser. 

testAllProps('boxSizing')  // true
testAllProps('display', 'block') // true
testAllProps('display', 'penguin') // false

.testProp only tests against non-prefix version.

9.9. UAParser.js

Browser detect, engine, OS, CPU and device type/model from userAgent string. Git 

<script src="https://cdn.jsdelivr.net/npm/ua-parser-js@0/dist/ua-parser.min.js"></script>

var parser = new UAParser();
var r = parser.getResult(); // { ua: '', browser: {}, cpu: {}, device: {}, engine: {}, os: {} }
console.log(r.browser);     // {name: "Chromium", version: "15.0.874.106"}

// browser.name
// Amaya, Android Browser, Arora, Avant, Baidu, Blazer, Bolt, Bowser, Camino, Chimera, 
Chrome [WebView], Chromium, Comodo Dragon, Conkeror, Dillo, Dolphin, Doris, Edge, 
Epiphany, Fennec, Firebird, Firefox, Flock, GoBrowser, iCab, ICE Browser, IceApe, 
IceCat, IceDragon, Iceweasel, IE[Mobile], Iron, Jasmine, K-Meleon, Konqueror, Kindle, 
Links, Lunascape, Lynx, Maemo, Maxthon, Midori, Minimo, MIUI Browser, [Mobile] Safari, 
Mosaic, Mozilla, Netfront, Netscape, NetSurf, Nokia, OmniWeb, Opera [Mini/Mobi/Tablet], 
PhantomJS, Phoenix, Polaris, QQBrowser, QQBrowserLite, Quark, RockMelt, Silk, Skyfire, 
SeaMonkey, Sleipnir, SlimBrowser, Swiftfox, Tizen, UCBrowser, Vivaldi, w3m, Waterfox, 
WeChat, Yandex

r.device // { model: '', type: '', vendor: '' }

// model
// console, mobile, tablet, smarttv, wearable, embedded

9.10. flowpaper

Turn PDF into HTML5. https://flowpaper.com/download/

9.11. reveal.js

https://github.com/hakimel/reveal.js/

9.11.1. Plugins

All plugins

9.11.1.1. reveal.js-menu

Git clone and copy the whole folder 

Reveal.initialize({
        // ...

        dependencies: [
                // ... 

                { src: 'plugin/reveal.js-menu/menu.js' }
        ]
});

Add id="theme" if theme is used 

<link rel="stylesheet" href="css/theme/black.css" id="theme">

Add data-menu-title to section, if not, the first .menu-title is used and it doesn't have to be displayed. Or change option:titleSelector. If still no title and option:hideMissingTitles is set to false, then the slide will not be included in menu

Add custom item (panel) in menu top, default is slides and close button. 

Reveal.initialize({
        // ...

        menu: {
                // ...

                custom: [
                        { title: 'Links', icon: '<i class="fa fa-external-link">', src: 'links.html' },
                        { title: 'About', icon: '<i class="fa fa-info">', content: '<p>This slidedeck is created with reveal.js</p>' }
                ]
        }
});
src
when clicked, the external source is loaded into the menu content section.
content
any html but you can add your own menu items
<h1>Links</h1>
<ul class="slide-menu-items">
        <li class="slide-menu-item"><a href="#/transitions">Transitions</a></li>
        <li class="slide-menu-item"><a href="#/13">Code highlighting</a></li>
</ul>

// you can also link to any where
<h1>External Links</h1>
<ul class="slide-menu-items">
        <li class="slide-menu-item"><a href="https://github.com/denehyg/reveal.js-menu">Reveal.js-menu</a></li>
        <li class="slide-menu-item"><a href="https://github.com/hakimel/reveal.js">Reveal.js</a></li>
</ul>

Options and their default 

Reveal.initialize({
        // ...

        menu: {
          side: 'left', // or 'right',
          markers: true, // Add icon to menu title
        },
});
titleSelector: 'h1, h2, h3, h4, h5, h6'
e.g. 'p.lead'
(no term)
hideMissingTitles: false

9.13. BabylonJS

Install Blender
https://www.blender.org/
(no term)
File > User Preferences > Add-ons > search dxf and enable Export and Import AutoCAD DXF Format
(no term)
Or import "dxf ascii release 14 or under"
Install addon to export from .blend to .babylon
http://doc.babylonjs.com/resources/blender
(no term)
Export as .babylon
(no term)
Get free 3d .obj, .fbx, .3ds files from Free3d.com or Clara.io
(no term)
Tutorial: https://github.com/mpwassler/3dproductview
(no term)
Callback in runRenderLoop is run 60 times per second to draw a frame

9.13.1. Shapes

https://doc.babylonjs.com/babylon101/discover_basic_elements

var shape = BABYLON.MeshBuilder.Create[ShapeName](name, options, scene);

  • box, Sphere, Plane, Ground

9.14. Angular

9.14.2. 1.x AngularJS

9.14.2.1. Structure

https://ajax.googleapis.com/ajax/libs/angularjs/1.4.7/angular.js jquery common/common.module.js // angular.module('ajsNgComponents', []); common/config.js ng:config common/filters.js ng:filter common/fitlerableCategories.js ng:factory

9.14.2.2. Calling Order

app.config() app.run() directive's compile functions (if exist) app.controller() directive's link functions

9.14.2.3. Config ng:config

ng:html5mode 

angular
  .module('ajsNgComponents')
  .config(locatgionProviderConfig)
  // .config is to run a Provider function
  // ng:provider
  .value('layout','boxed_fullwidth')
  .value('layoutClass', {
    boxed_fullwidth: 'col-lg-3 col-md-4 col-sm-6 col-xs-12',
    boxed_left: 'col-lg-4 col-md-6 col-sm-6 col-xs-12',
    boxed_right: 'col-lg-4 col-md-6 col-sm-6 col-xs-12'
  });

// Enable html5mode
locatgionProviderConfig.$inject = ['$locationProvider'];
function locatgionProviderConfig($locationProvider) {
  $locationProvider.html5Mode({
    enabled: true,
    requireBase: false
  });
}
9.14.2.4. Run

Run injection is a bit different 

app.run(ajsRun);
// Not app.run('ajsRun', ajsRun);

ajsRun.$inject = ['$rootScope'];
function ajsRun($rootScope) {

}
9.14.2.5. Filter ng:filter
angular
  .module('ajsNgComponents')
  .filter('singleListingFilter', singleListingFilter)
  .filter('item_category_value', item_category_value)
  .filter('ignore_empty_value', ignore_empty_value)
  .filter('translate', translate);

// $scope can't be injected to filter
// Instead, pass scope.varname as a param into filter.
// Or inject $rootScope

// In view
/*
<tr ng-repeat="detail in details | ignore_empty_value:aListing"
 ng-init="detail = aListing.post_meta.options[detailKey]">
  <td ng-bind="(detail.name | translate:detail)"></td>
  <td ng-bind="(detail.value | translate)"></td>
</tr>
*/

function singleListingFilter() {
  return function(items,var1) {
    var r = [];
    for (var i= 0, len=items.length; i< len; i++) {
  if (items[i].id == var1){
    r.push(items[i]);
    break;
  }
    }
    return r;
  };
}

// <td ng-bind="((x.options | item_category_value:detail:lwp_options) | translate)"></td>
function item_category_value() {
  return function(options, category, lwp_options) {
    var value = "";
    return value;
  };
}

function ignore_empty_value() {
  return function(items, aListing) {
    var r = [];
    angular.forEach(items, function(item) {
  if (aListing.post_meta.options[item].value) {
    r.push(item);
  }
    });
    return r;
  };
}

translate.$inject = ['$rootScope'];
function translate($rootScope) {
  return function(word, languages) {
    var r = word;
    if (typeof $rootScope.lang !== "undefined") {
  if ( typeof languages !== "undefined"
    && typeof languages['lang'] !== "undefined"
    && typeof languages['lang'][$rootScope.lang] !== "undefined") {
    r = languages['lang'][$rootScope.lang];
  }
  else {
    r = $rootScope.translate_text(word);
  }
    }
    return r;
  };
}
9.14.2.6. Factory ng:factory

Factory is better than app.value(); 

(function() {
  'use strict';
  angular
    .module('ajsNgComponents')
    .factory('filterableCategories', filterableCategories)
    .factory('clientId', clientIdFactory)
    .factory('apiToken', apiTokenFactory);

  // Inject $rootScope so that any method defined here can use
  filterableCategories.$inject = ['$rootScope'];
  function filterableCategories($rootScope) {
    return {
  getFilters: getFilters,
  getListingsFilterObjByURL: getListingsFilterObjByURL,
  getSelectedFiltersObjByURL: getSelectedFiltersObjByURL,
  getSearchInventoryDropdown: getSearchInventoryDropdown
    };

    function getFilters(filterable_categories, lwp_options) {
  // $rootScope can be used here
  var r = [];
  return r;
    }
  }

  function clientIdFactory() {
    return '123';
  }

  apiTokenFactory.$inject = ['clientId'];
  function apiTokenFactory(clientId) {
    var encrypt = function(data1, data2) {
  return data1+data2;
    }
    var secret = window.localStorage.getItem('secret');
    var apiToken = encrypt(clientId, secret);
    return apiToken;
  }

})();
9.14.2.7. Service ng:service
app.service('unicornLauncher', UnicornLauncherService);
UnicornLauncherService.$inject = ['apiToken'];
function UnicornLauncherService(apiToken) {
  this.launchedCount = 0;
  this.launch = function () {
    // Use apiToken to do something
    this.launchedCount++;
  }
}
9.14.2.8. Provider ng:provider

Provider is a service and it's the only thing can be inserted to .config

Instantiation happens in config 

app.config(myProviderConfig);
myProviderConfig.$inject = ['myProviderProvider'];
function myProviderConfig(myProviderProvider){
  myProviderProvider.thingFromConfig = 'This was set in config';
});

Define myProvider 

app.provider('myProvider', function(){
 // app.config() can only access the next 2 lines
 this._artist = '';
 this.thingFromConfig = '';

 this.$get = function(){
   var that = this;

   return {
 getArtist: function(){
   return that._artist;
 },
 thingOnConfig: that.thingFromConfig,
 unicornLauncher: newUnicornLauncher
   }

   newUnicornLauncher.$inject = ['apiToken'];
   function newUnicornLaunder(apiToken) {
 return new UnicornLauncherService(apiToken);
   }

 }
});

app.controller('myProvider', function($scope, myProvider){
  $scope.artist = myProvider.getArtist();
  $scope.data.thingFromConfig = myProvider.thingOnConfig;
});
9.14.2.9. Change URL without reloading page

First enable ng:html5mode

inject $location and $window 

myNewUrl = '/path/?p=1&q=2';
$location.url(myNewUrl);
$location.replace();
$window.history.pushState(null, 'any', $location.absUrl());

9.14.3. 2.x Angular

9.14.3.1. QuickStart seed
  1. Install

    Local dev env which is the same as Online playground https://angular.io/guide/setup 

    git clone https://github.com/angular/quickstart.git quickstart
cd quickstart
npm install
npm start
# Default port is localhost:3000

    Backup .gitignore

    And remove git-related artifacts 

    # OS/X
xargs rm -rf < non-essential-files.osx.txt
rm src/app/*.spec*.ts
rm non-essential-files.osx.txt

# Windows cmd
for /f %i in (non-essential-files.txt) do del %i /F /S /Q
rd .git /s /q
rd e2e /s /q
  2. .gitignore
    .idea
node_modules

jspm_packages
npm-debug.log
debug.log
src/**/*.js
!src/systemjs.config.extras.js
!src/systemjs.config.js
!src/systemjs-angular-loader.js
*.js.map
e2e/**/*.js
e2e/**/*.js.map
_test-output
_temp
  3. Files

    https://angular.io/guide/setup-systemjs-anatomy

    src/main.ts It's loaded in index.html as in System.import('main.js').catch(function(err){ console.error(err); }); Refer to ng:bootstrap

    src/systemjs.config.js It's loaded in index.html. Tells systemJS module loader where to find modules referenced in JS import statements. e.g. import { Component } from '@angular/core'

    src/tsconfig.json :: how to transpile TypeScript files into JS files

    src/app/app.module.ts :: ng:root module src/app/app.component.ts :: tree of components ng:component

    package.json tslint.json :: Inspect TypeScript code and complains when there's violation bs-config.json :: lite-server BrowserSync setting

9.14.3.2. SystemJS vs Webpack

Quickstart uses SystemJS to dynamically load files on demand on the client side. It's also called lazy load. Angular CLI uses Webpack to prepare files. Also does file minifying, transpilation (e.g. from SASS to CSS).

9.14.3.3. package.json

https://angular.io/guide/npm-packages

  1. devDependencies
    "devDependencies": {
    "concurrently": "^3.2.0",
    "lite-server": "^2.2.2", // static file server
    "typescript": "~2.1.0", // tsc TypeScript compiler

    "canonical-path": "0.0.2",
    "tslint": "^3.15.1",
    "lodash": "^4.16.4",
    "jasmine-core": "~2.4.1",
    "karma": "^1.3.0",
    "karma-chrome-launcher": "^2.0.0",
    "karma-cli": "^1.0.1",
    "karma-jasmine": "^1.0.2",
    "karma-jasmine-html-reporter": "^0.2.2",
    "protractor": "~4.0.14",
    "rimraf": "^2.5.4",

    "@types/node": "^6.0.46",
    "@types/jasmine": "2.5.36"
}
  2. dependencies
    "dependencies": {
    "@angular/common": "~4.0.0", // common services, pipes, and directives provided by Angular
    "@angular/compiler": "~4.0.0", // Angular Template Compiler
    "@angular/core": "~4.0.0", // e.g. all metadata decorators, Component, Directive, dependency injection and component lifecycle hooks
    "@angular/forms": "~4.0.0",
    "@angular/http": "~4.0.0", // Angular's HTTP client.
    "@angular/platform-browser": "~4.0.0", // DOM and browser related. Native directives, pipes, etc. Help render into DOM.
    "@angular/platform-browser-dynamic": "~4.0.0", 
    // Include Providers and a bootstrap method for apps that compile templates on the client.
    // Don't use offline compilation, use this for bootstrapping during dev and for bootstrapping plunker samples
    "@angular/router": "~4.0.0", // Component router

    "angular-in-memory-web-api": "~0.3.0", // Angular simulates a remote server's web api without requiring an actual server or real HTTP calls.
    "systemjs": "0.19.40", // A dynamic module loader compatible with ES2015 module specs. Alternative webpack
    "core-js": "^2.4.1", // polyfill
    "rxjs": "5.0.1", // polyfill
    "zone.js": "^0.8.4" // polyfill
}
  3. scripts
    "scripts": {
    "build": "tsc -p src/",
    "build:watch": "tsc -p src/ -w",
    "build:e2e": "tsc -p e2e/",
    "serve": "lite-server -c=bs-config.json",
    "serve:e2e": "lite-server -c=bs-config.e2e.json",
    "prestart": "npm run build",
    "start": "concurrently \"npm run build:watch\" \"npm run serve\"",
    "pree2e": "npm run build:e2e",
    "e2e": "concurrently \"npm run serve:e2e\" \"npm run protractor\" --kill-others --success first",
    "preprotractor": "webdriver-manager update",
    "protractor": "protractor protractor.config.js",
    "pretest": "npm run build",
    "test": "concurrently \"npm run build:watch\" \"karma start karma.conf.js\"",
    "pretest:once": "npm run build",
    "test:once": "karma start karma.conf.js --single-run",
    "lint": "tslint ./src/**/*.ts -t verbose"
}
9.14.3.4. tsconfig.json, d.ts, lib.d.ts

https://angular.io/guide/typescript-configuration http://www.typescriptlang.org/docs/handbook/tsconfig-json.html

Quickstart src/tsconfig.json 

{
  "compilerOptions": {
    "target": "es5",
    "module": "commonjs",
    "moduleResolution": "node",
    "sourceMap": true,
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "lib": [ "es2015", "dom" ],
    "noImplicitAny": true,
    "suppressImplicitAnyIndexErrors": true
  }
}

Angular CLI src/tsconfig.app.json extends ./tsconfig.json. See ./.angular-cli.json

d.ts file is a TypeScript Declaration file, which tells the compiler what libraries you load.

Libraries, such as jQuery and Angular extend the JavaScript environment with features and syntax. node_modules/@angular/core/ folder contains several d.ts files. You don't need to do anything to get these typing files for Angular library packages.

lib.XXX.d.ts Typescript includes many lib.XXX.d.ts files for you to load ambient declarations. In tsconfig.json for Quickstart, there's a line to load lib.es2015.d.ts and lib.dom.d.ts Based on the --target, TypeScript adds other declarations automatically like Promose if the target is es6 

"lib": ["es2015", "dom"]

Since many libraries, e.g. jQuery, Jasmine and Lodash, do not include d.ts files in npm packages.

npm install these scoped packages @types/* and Typescript automatically recognizes them. Quickstart dependencies have 2: @types/node @types/jasmine

9.14.3.5. lite-server npm:lite-server
9.14.3.6. CLI

https://github.com/angular/angular-cli https://github.com/angular/angular-cli/wiki

  1. Install & Update
    # Install or update
npm install -g @angular/cli

    Update :: update both the global package and your project's local package. 

    # Global package:

npm uninstall -g @angular/cli
npm cache clean
npm install -g @angular/cli@latest
Local project package:

rm -rf node_modules dist 
# use this in Windows Command Prompt
# rmdir /S/Q node_modules dist
# use this in Windows PowerShell
# rm -r -fo node_modules,dist
npm install --save-dev @angular/cli@latest
npm install
  2. Generate Component, Directive, Pipes and Services

    CLI adds reference in app.module.ts automatically. 

    # ng generate
ng g component my-new-component
ng g component ../newer-cmp
ng g component feature/new-cmp
ng g directive new-directive
ng g pipe new-pipe
ng g service new-service
ng g class new-class
ng g guard new-guard
ng g interface new-interface
ng g enum new-enum
ng g module new-module
  3. New project
    ng new my-project
cd my-project
ng serve

    src/index.html doesn't include any javascript files while quickstart includes some.

  4. build

    ng build --prod :: Delete and add files to ./dist

    All commands that build or serve your project, ng build/serve/e2e, will delete the output directory (dist/ by default). This can be disabled via the --no-delete-output-path (or --delete-output-path=false) flag.

  5. eject ng:cli:webpack

    Do this after you have stopped ng serve. This will copy the webpack configuration and modify .angular-cli.json and package.json. Do git status --ignored to see the webpack configurations. After ng eject, ng commands will not work. See changes in package.json. Such as: instead of ng serve, do npm run build & npm run start You can now modify webpack configuration with real effects.

    ng eject --prod production environment. It includes UglifyJsPlugin npm:webpack

    Bundle files are main.ts, polyfills.ts and styles.css

9.14.3.7. Polyfills

Other polyfills 

<script src="node_modules/core-js/client/shim.min.js"></script>
9.14.3.8. Bootstrap ng:bootstrap

Create a browser platform for dynamic compilation (JIT) and bootstraps the AppModule by referring to the root module app.module.ts Refer to ng:root module 

import { platformBrowserDynamic } from '@angular/platform-browser-dynamic';

import { AppModule } from './app/app.module';

platformBrowserDynamic().bootstrapModule(AppModule);
9.14.3.9. Root Module ng:root module
import { NgModule }      from '@angular/core';
import { BrowserModule } from '@angular/platform-browser';
import { AppComponent }  from './app.component';
// import { FormsModule } from '@angular/forms';

// This decorator makes AppModule as an Angular module class (NgModule class).
// a metadata object to tell Angular how to compile and launch the app
@NgModule({
  // Only put NgModule classes in imports array
  imports:      [ BrowserModule
    // , FormsModule
  ],
  // Only declarables - components, directives and pipes - belong in declarations array. Not NgModule classes
  declarations: [ AppComponent ],
  // Create components listed in the bootstrap array and insert each one into browser DOM
  // It's common to only include one tree of components, which is one file: app.component.ts
  bootstrap:    [ AppComponent ]
})
export class AppModule { }
9.14.3.10. ngModule

https://angular.io/guide/ngmodule

9.14.3.11. Component ng:component
  • Create a new html element. It's also a directive with a template
  • Refer to ng:api:component

src/app/app.component.ts 

import { Component } from '@angular/core';

// Decorator
@Component({
  selector: 'my-app',
  template: `<h1>Hello {{name}}</h1>`,
  inputs: ['inputVar1'],  // Use this or @Input. Refer to ng:input output
  outputs: ['outputVar1'], // Use this or @Output. Refer to ng:input output
})
export class AppComponent  { name = 'Angular'; }
9.14.3.12. Template
  1. Files

    src/app/[component-name]/[component-name].component.html src/app/[component-name]/[component-name].component.css src/app/[component-name]/[component-name].component.ts

    src/app/heroes/heroes.component.ts Use component-relative URLs, prefixed with ./ 

    @Component({
  selector: 'toh-heroes',
  templateUrl: './heroes.component.html', // template or templateUrl, can't be both
  styleUrls:  ['./heroes.component.css'] // each css file is independent! Don't worry about CSS conflicts
  // , styles: [ ".btn {background-color: green;}", ".btn:hover {background-color: pink;}"]
})

export class HeroesComponent implements OnInit {
  /* name = 'abc'; 
 artists: string[]; // could be any
  */
  /* or 
 name: string;
 constructor() {
       this.name = 'abc';
       this.artists = ["1", "2"]
 }
  */

  heroes: Observable<Hero[]>;
  selectedHero: Hero;

 onClick(e) {
  // event
  // console.log(e);
  this.name = e.target.innerHTML;
 }

 addArtist(v) {
   if (v!=='') {
 this.artists.push({
       name: v,
       school: 'abc'
 });
   } 
 } 

 constructor(private heroService: HeroService) { }

  ngOnInit() {
    this.heroes = this.heroService.getHeroes();
  }
}

    src/app/heroes/heroes.component.html 

    <div>
  <h2>My Heroes</h2>
  <ul class="heroes">
    <li *ngFor="let hero of heroes | async" (click)="selectedHero=hero">
      <span class="badge">{{hero.id}}</span> {{hero.name}}
    </li>
  </ul>
  <div *ngIf="selectedHero">
    <h2>{{selectedHero.name | uppercase}} is my hero</h2>
  </div>
</div>
  2. {{ }}

    Nonsupported in {{ }}

    • Assignments
    • Newing up variables
    • Chaining expressions
    • Incrementing/decrementing

    A function can be run 

    <div>{{ wasWatched() }}</div>

export class MediaItemComponent { 
  wasWatched() {
    return true;
  }
}
  3. Styles

    Module bundler like Webpack can load sytles from external files at build time. styles: [require('my.component.css')]

    Use :host to target the host element (one element)

    :host-context() looks for a CSS class in any ancestor of the component host element, up to the document root. The :host-context() selector is useful when combined with another selector.

    This applies a background-color style to all <h2> elements inside the component, only if some ancestor element has the CSS class theme-light. 

    :host-context(.theme-light) h2 {
  background-color: #eef;
}

    /deep/ or >>> works to any depth of nested components, and it applies to both the view children and content children of the component.

    This targets all <h3> elements, from the host element down through this component to all of its child elements in the DOM. 

    :host /deep/ h3 {
  font-style: italic;
}

    Use /deep/ and >>> selectors only with emulated view encapsulation. Emulated is the default and most commonly used view encapsulation. Refer to ng:view encapsulation

    Use <link> in template, path is relative to the root. 

    @Component({
  selector: 'hero-team',
  template: `
    <link rel="stylesheet" href="app/hero-team.component.css">
    <h3>Team</h3>
    <ul>
      <li *ngFor="let member of hero.team">
        {{member}}
      </li>
    </ul>`
})

    Use @imports inside my.component.css 

    @import 'hero-details-box.css';

    ng:view encapsulation Default is ViewEncapsulation.Emulated, rename CSS code to scope the CSS to the component's view. ViewEncapsulation.None adds CSS to the global styles. ViewEncapsulation.Native uses browser's native shadow DOM to attach a shadow DOM to the component's host element.

    If the encapsulation is set to ViewEncapsulation.Emulated and the component has no styles nor styleUrls, the encapsulation will automatically be switched to ViewEncapsulation.None. 

    @Component({
  selector: 'toh-heroes',
  templateUrl: './heroes.component.html',
  styleUrls:  ['./heroes.component.css'],
  encapsulation: ViewEncapsulation.Emulated
})
  4. Event (eventType)

    https://angular.io/guide/template-syntax#event-binding 

    (click)="onClick($event)"
(input)="name=$event.target.value"
(keyup.enter)="addArtist(newArtist.value); newArtist.value=''"

    $event is a DOM event object

  5. Template Reference variable #var
    <input #newArtist (keyup.enter)="addArtist(newArtist.value); newArtist.value=''">
<button (click)="addArtist(newArtist.value); newArtist.value=''">Add</button>

    #newArtist can be used anywhere in the template.

  6. One-way binding
    // Instead of
<lable>Search <span *ngIf="name">for: {{ name }}</span></label>

// Use
<lable>Search <span *ngIf="name" [innerHTML]="' for: ' + name"></span></lable>
// [innerHTML] can be bind-innerHTML

// Or use
<span innerHTML="{{ ' for: ' + name }}"

// DOM element attribute
<img [src]="iconUrl"/>

// Bind clicked DOM element to a variable
<ul class="heroes">
  <li #selectedHero *ngFor="let hero of heroes | async" (click)="onClick(hero, selectedHero)">
    <span class="badge">{{hero.id}}</span> {{hero.name}}
  </li>
</ul>

// inside export class
onClick(item, myElement) {
  this.name = item.name;
  myElement.style.backgroundColor="red";
}
  7. Two-way binding [(x)] [(ngModel)] [ngModel] (ngModelChange)
    <input #newArtist 
[value]="name"
(input)="name=$event.target.value"
(keyup.enter)="addArtist(newArtist.value); newArtist.value=''">

// It's equivalent to
<input #newArtist 
[(ngModel)]="name"
(keyup.enter)="addArtist(newArtist.value); newArtist.value=''">

    <input
  [ngModel]="currentHero.name"
  (ngModelChange)="setUppercaseName($event)">

    ngModel needs ng:FormsModule

  8. @Input, @Output ng:input output

    media-item.component.ts 

    import { Component, Input } from '@angular/core';

@Component({
    selector: 'mw-media-item',
    templateUrl: './media-item.component.html',
    styleUrls: ['./media-item.component.css']
})
export class MediaItemComponent {
    // @Input('mediaItemToWatch') mediaItem;
    // The mediaItemToWatch is an alias of mediaItem
    // It's not recommended to have alias
    @Input() mediaItem;

    // strict typing
    @Input()  size: number | string;

    @Output() delete = new EventEmitter();
    // Notice delete is defined in sub component and exists inside mw-media-item DOM element

    onDelete() {
       // emit an event to the parent app.component.ts
       this.delete.emit(this.mediaItem);
    }

}

    app.component.ts 

    export class AppComponent {
  onMediaItemDelete(mediaItem) {

  }

  firstMediaItem = {
    id: 1,
    name: "Firebug",
    medium: "Series",
    category: "Science Fiction",
    year: 2010,
    watchedOn: 1294166565384,
    isFavorite: false
  };
}

    mediaItem can be used as a new property for binding [mediaItem] or {{mediaItem}} app.component.html 

    <section>
  <header>
    <h1>Media Watch List</h1>
    <p class="description">Keeping track of the media I want to watch.</p>
  </header>
  <mw-media-item 
    [mediaItem]="firstMediaItem"
    (delete)="onMediaItemDelete($event)"></mw-media-item>
</section>

    media-item.component.html 

    <h2>{{ mediaItem.name }}</h2>
<div>Watched on {{ mediaItem.watchedOn }}</div>
<div>{{ mediaItem.category }}</div>
<div>{{ mediaItem.year }}</div>
<div class="tools">
  <a class="delete" (click)="onDelete()">
    remove
  </a>
  <a class="details">
    watch
  </a>
</div>
  9. Structural directives (builtin) ngIf ngFor ngSwitch <ng-container>
    <div *ngIf="mediaItem.watchedOn">Watched on {{ mediaItem.watchedOn }} </div>

    Even though ngIf is true, only <div> will display 

    <template [ngIf]="mediaItem.watchedOn">
  <div>Watched on {{ mediaItem.watchedOn }} </div>
</template>

    ngFor :: refer to ng:pipe

    ngSwitch 

    <div [ngSwitch]="hero?.emotion">
  <happy-hero    *ngSwitchCase="'happy'"    [hero]="hero"></happy-hero>
  <sad-hero      *ngSwitchCase="'sad'"      [hero]="hero"></sad-hero>
  <confused-hero *ngSwitchCase="'confused'" [hero]="hero"></confused-hero>
  <unknown-hero  *ngSwitchDefault           [hero]="hero"></unknown-hero>
</div>

    Only one structural directive can be applied to one host element

    <ng-container> This is almost works like <template> 

    <p>
  I turned the corner
  <span *ngIf="hero">
    and saw {{hero.name}}. I waved
  </span>
  and continued on my way.
</p>

    In <select>, before we do and dropdown is empty because a browser won't display an <option> within a <span> 

    <div>
  Pick your favorite hero
  (<label><input type="checkbox" checked (change)="showSad = !showSad">show sad</label>)
</div>
<select [(ngModel)]="hero">
  <span *ngFor="let h of heroes">
    <span *ngIf="showSad || h.emotion !== 'sad'">
      <option [ngValue]="h">{{h.name}} ({{h.emotion}})</option>
    </span>
  </span>
</select>

    <div>
  Pick your favorite hero
  (<label><input type="checkbox" checked (change)="showSad = !showSad">show sad</label>)
</div>
<select [(ngModel)]="hero">
  <ng-container *ngFor="let h of heroes">
    <ng-container *ngIf="showSad || h.emotion !== 'sad'">
      <option [ngValue]="h">{{h.name}} ({{h.emotion}})</option>
    </ng-container>
  </ng-container>
</select>
  10. Attribute directives (builtin) ngClass ngStyle

    media-list.component.html 

    <section>
  <mw-media-item
    [ngClass]="{'medium-movies': mediaItem.medium==='Movies', 'medium-series':mediaItem.medium==='Series'}" 
    *ngFor="let mediaItem of mediaItems"
    [mediaItem]="mediaItem"
    (delete)="onMediaItemDelete($event)"></mw-media-item>
</section>

    [class] or [style] with a string can also be used 

    <!-- Reset class to badCurly only -->
<div class="bad curly special"
 [class]="badCurly">Bad curly</div>

<!-- toggle the "special" class on/off with a property -->
<div [class.special]="isSpecial">The class binding is special</div>


<button [style.color]="isSpecial ? 'red': 'green'">Red</button>
<button [style.background-color]="canSave ? 'cyan': 'grey'" >Save</button>
<button [style.font-size.em]="isSpecial ? 3 : 1" >Big</button>
<button [style.font-size.%]="!isSpecial ? 150 : 50" >Small</button>
9.14.3.13. Pipe ng:pipe
  1. Builtin pipes

    https://angular.io/api?query=pipe

    uppercase, lowercase, titlecase

    1. Date

      date_expression | date[:format] format 'medium': equivalent to 'yMMMdjms' (e.g. Sep 3, 2010, 12:05:08 PM for en-US) 'short': equivalent to 'yMdjm' (e.g. 9/3/2010, 12:05 PM for en-US) 'fullDate': equivalent to 'yMMMMEEEEd' (e.g. Friday, September 3, 2010 for en-US) 'longDate': equivalent to 'yMMMMd' (e.g. September 3, 2010 for en-US) 'mediumDate': equivalent to 'yMMMd' (e.g. Sep 3, 2010 for en-US) 'shortDate': equivalent to 'yMd' (e.g. 9/3/2010 for en-US) 'mediumTime': equivalent to 'jms' (e.g. 12:05:08 PM for en-US) 'shortTime': equivalent to 'jm' (e.g. 12:05 PM for en-US) 

      <p>Today is {{today | date}}</p>
<p>Or if you prefer, {{today | date:'fullDate'}}</p>
<p>The time is {{today | date:'jmZ'}}</p>
    2. slice

      slice:start[:end] 

      <h2>{{ mediaItem.name | slice:0:10 }}</h2>
  2. Custom pipe

    https://angular.io/guide/pipes

    src/app/search.pipe.ts 

    import { Pipe, PipeTransform } from '@angular/core';

@Pipe({
  name: 'search',
  // Default is pure, which means the pipe doesn't change the data
  // pure: true,
})

export class SearchPipe implements PipeTransform {
  // pipe is used in ngFor, the first parameter is the pipe data without specifying in template
  transform(pipeData, pipeModifier) {
    return  pipeData.filter((eachItem)=> {
      return eachItem['name'].toLowerCase().includes(pipeModifier.toLowerCase()) 
          || eachItem['reknown'].toLowerCase().includes(pipeModifier.toLowerCase());
    });
  }

  // Another example for ngFor pipe. It returns an array
  transform(mediaItems) {
    var categories = [];
    mediaItems.forEach(mediaItem => {
      if (categories.indexOf(mediaItem.category) <= -1) {
        categories.push(mediaItem.category);
      }
    });
    return categories.join(', ');
  }

 /* Another transform interface example */
 /*
 transform(value: number, exponent: string): number {
    let exp = parseFloat(exponent);
    return Math.pow(value, isNaN(exp) ? 1 : exp);
 }
 */

}

    app.html 

    <ul class="artistlist cf"
  *ngIf="query">
  <li class="artistlist-item cf"
    (click)="showArtist(item); query=''"
    *ngFor="let item of (artists | search: query)">
    <artist-item class="content" [artist]=item></artist-item>
  </li>
</ul>
9.14.3.14. Directive ng:directive
  1. Custom attribute directive HostListener HostBinding

    It's better to add prefix 'my' to selector name hightlight.directive.ts 

    import { Directive, ElementRef, HostListener, Input } from '@angular/core';

@Directive({
  selector: '[myHighlight]'
})
export class HighlightDirective {

  constructor(private el: ElementRef) { }

  @Input() defaultColor: string;

  @Input('myHighlight') highlightColor: string;

  @HostListener('mouseenter') onMouseEnter() {
    this.highlight(this.highlightColor || this.defaultColor || 'red');
  }

  @HostListener('mouseleave') onMouseLeave() {
    this.highlight(null);
  }

  private highlight(color: string) {
    this.el.nativeElement.style.backgroundColor = color;
  }
}

    app.component.html 

    <h1>My First Attribute Directive</h1>

<h4>Pick a highlight color</h4>
<div>
  <input type="radio" name="colors" (click)="color='lightgreen'">Green
  <input type="radio" name="colors" (click)="color='yellow'">Yellow
  <input type="radio" name="colors" (click)="color='cyan'">Cyan
</div>
<p [myHighlight]="color">Highlight me!</p>

<p [myHighlight]="color" defaultColor="violet">
  Highlight me too!
</p>

<hr>
<p><i>Mouse over the following lines to see fixed highlights</i></p>

<p [myHighlight]="'yellow'">Highlighted in yellow</p>
<p myHighlight="orange">Highlighted in orange</p>

    Another example using HostBinding which binds host (DOM) property to a getter or another property favorite.directive.ts 

    import { Directive, HostBinding, Input } from '@angular/core';

@Directive({
  selector: '[mwFavorite]'
})
export class FavoriteDirective {
  @HostBinding('class.is-favorite') isFavorite; 
  // Use ~isFavorite = true~ to set a default
  @Input() set mwFavorite(value) {
    this.isFavorite = value;
  }
  /*
 @HostBinding('attr.something') get something() { 
    return this.somethingElse; 
 }
 */
}

    media-item.component.html 

    <h2>{{ mediaItem.name }}</h2>
<template [ngIf]="mediaItem.watchedOn">
  <div>Watched on {{ mediaItem.watchedOn }}</div>
</template>
<div>{{ mediaItem.category }}</div>
<div>{{ mediaItem.year }}</div>
<div class="tools">
  <svg
    [mwFavorite]="mediaItem.isFavorite"
    class="favorite" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24">
    <path d="M12 9.229c.234-1.12 1.547-6.229 5.382-6.229 2.22 0 4.618 1.551 4.618 5.003 0 3.907-3.627 8.47-10 12.629-6.373-4.159-10-8.722-10-12.629 0-3.484 2.369-5.005 4.577-5.005 3.923 0 5.145 5.126 5.423 6.231zm-12-1.226c0 4.068 3.06 9.481 12 14.997 8.94-5.516 12-10.929 12-14.997 0-7.962-9.648-9.028-12-3.737-2.338-5.262-12-4.27-12 3.737z"
    />
  </svg>
  <a class="delete" (click)="onDelete()">
    remove
  </a>
  <a class="details">
    watch
  </a>
</div>
9.14.3.15. Model and Custom Component

src/app/app.component.ts 

import { Component } from '@angular/core';
import { ArtistItemComponent } from './artists/artist-item/artist-item.component';

// Model
export class Artist {
  name: string,
  shortname: string,
  reknown: string,
  bio: string
}

@Component({
  selector: 'my-app',
  templateUrl: './app.component.html',
  styleUrls: ['./app.component.css']
})
export class AppComponent { 
  artists = ARTISTS;
  currentArtist: Artist;
}

var ARTISTS: Artist[] = [...]

src/app/app.component.html 

<div class="card search">
  <h1 class="search-headline">Artist Directory</h1>
  <label class="search-label">search
    <span *ngIf="query"
      [innerHTML]="' for: ' + query"></span></label>
    <input class="search-input"
      [(ngModel)]="query" placeholder="type in search term here">
</div><!-- card search -->

<ul class="artistlist cf">
  <li class="artistlist-item cf"
    *ngFor="let item of artists">
    <artist-item class="content" [artist]=item></artist-item>
  </li>
</ul>

src/app/app.module.ts 

import { NgModule } from '@angular/core';
import { BrowserModule } from '@angular/platform-browser';
import { FormsModule } from '@angular/forms';

import { AppComponent } from './app.component';
import { ArtistItemComponent } from './artists/artist-item/artist-item.component';

@NgModule({
  imports: [
    BrowserModule, FormsModule
  ],
  declarations: [
    AppComponent, ArtistItemComponent
  ],
  bootstrap: [
    AppComponent
  ]
})

export class AppModule {}

src/app/artists/artist-item/artist-item.component.ts 

import { Component } from '@angular/core';

@Component({
  selector: 'artist-item',
  templateUrl: './artist-item.component.html',
  styleUrls: ['./artist-item.component.css'],
  inputs: ['artist']
})

export class ArtistItemComponent {}

src/app/artists/artist-item/artist-item.component.html 

<img class="artist-img" 
  src="images/{{artist.shortname}}_tn.jpg"
  alt="{{artist.name}} photo">
<div class="artist-info">
  <h2 class="artist-name">{{ artist.name }}</h2>
  <h3 class="artist-reknown">{{ artist.reknown }}</h3>
</div>
9.14.3.16. Form
  1. Template driven ngModel ngSubmit

    The form element has a name, then directly use ngModel Template driven form requires ng:FormsModule https://github.com/coursefiles/angular2-essential-training/blob/04_03b/app

    https://github.com/coursefiles/angular2-essential-training/blob/04_03b/app/app.module.ts

    src/app/media-item-form.component.ts 

    import { Component } from '@angular/core';

@Component({
  selector: 'mw-media-item-form',
  templateUrl: 'app/media-item-form.component.html',
  styleUrls: ['app/media-item-form.component.css']
})
export class MediaItemFormComponent {
  onSubmit(mediaItem) {
    console.log(mediaItem);
  }
}

    src/app/media-item-form.component.html 

    <header>
  <h2>Add Media to Watch</h2>
</header>
<form
  #mediaItemForm="ngForm"
  (ngSubmit)="onSubmit(mediaItemForm.value)">
  <ul>
    <li>
      <label for="medium">Medium</label>
      <select name="medium" id="medium" ngModel>
        <option value="Movies">Movies</option>
        <option value="Series">Series</option>
      </select>
    </li>
    <li>
      <label for="name">Name</label>
      <input type="text" name="name" id="name" ngModel>
    </li>
    <li>
      <label for="category">Category</label>
      <select name="category" id="category" ngModel>
        <option value="Action">Action</option>
        <option value="Science Fiction">Science Fiction</option>
        <option value="Comedy">Comedy</option>
        <option value="Drama">Drama</option>
        <option value="Horror">Horror</option>
        <option value="Romance">Romance</option>
      </select>
    </li>
    <li>
      <label for="year">Year</label>
      <input type="text" name="year" id="year" maxlength="4" ngModel>
    </li>
  </ul>
  <button type="submit">Save</button>
</form>
  2. Model driven, Validation

    Model driven form requires ng:ReactiveFormsModule instead of FormsModule. Both can be loaded at the same time

    ng:FormGroup ng:FormControl ng:Validators

    https://github.com/coursefiles/angular2-essential-training/tree/04_04b/app

    https://github.com/coursefiles/angular2-essential-training/blob/04_04b/app/app.module.ts

    media-item-form.component.ts 

    import { Component } from '@angular/core';
import { FormGroup, FormControl, Validators } from '@angular/forms';

@Component({
  selector: 'mw-media-item-form',
  templateUrl: 'app/media-item-form.component.html',
  styleUrls: ['app/media-item-form.component.css']
})
export class MediaItemFormComponent {
  form;

  ngOnInit() {
    this.form = new FormGroup({
      medium: new FormControl('Movies'),
      name: new FormControl('', Validators.compose([
        Validators.required, // this field has to pass
        Validators.pattern('[\\w\\-\\s\\/]+')
      ])), // builtin validator
      /* if it's wrong, DOM will have ng-invalid css class but it's not mandatory
      name: new FormControl('', Validators.pattern('[\\w\\-\\s\\/]+')),
      */
      category: new FormControl(''),
      year: new FormControl('', this.yearValidator),
    });
  }  

  yearValidator(control) {
    if (control.value.trim().length === 0) {
      return null;
    }
    let year = parseInt(control.value);
    let minYear = 1800;
    let maxYear = 2500;
    if (year >= minYear && year <= maxYear) {
      return null;
    } else {
      return {
        'year': {
          min: minYear,
          max: maxYear
        }
      };
    }
  }

  onSubmit(mediaItem) {
    console.log(mediaItem);
  }
}

    media-item-form.component.html 

    <form
  [formGroup]="form"
  (ngSubmit)="onSubmit(form.value)">
  <ul>
    <li>
      <label for="medium">Medium</label>
      <select name="medium" id="medium" formControlName="medium">
        <option value="Movies">Movies</option>
        <option value="Series">Series</option>
      </select>
    </li>
    <li>
      <label for="name">Name</label>
      <input type="text" name="name" id="name" formControlName="name">
      <div *ngIf="form.controls.name.errors?.pattern" class="error">
        Name has invalid characters
      </div>
    </li>
    <li>
      <label for="category">Category</label>
      <select name="category" id="category" formControlName="category">
        <option value="Action">Action</option>
        <option value="Science Fiction">Science Fiction</option>
        <option value="Comedy">Comedy</option>
        <option value="Drama">Drama</option>
        <option value="Horror">Horror</option>
        <option value="Romance">Romance</option>
      </select>
    </li>
    <li>
      <label for="year">Year</label>
      <input type="text" name="year" id="year" maxlength="4" formControlName="year">
      <div *ngIf="form.controls.year.errors?.year" class="error">
        Must be between 
        {{form.controls.year.errors?.year.min}} 
        and 
        {{form.controls.year.errors?.year.max}}
      </div>
    </li>
  </ul>
  <button type="submit" [disabled]="!form.valid">Save</button>
</form>
9.14.3.17. Service, Dependency Injection

Either use constructor or @Inject to include a service. Define services in a provider, when service is injected, if it has been created before, reuse it. If not, create it. Service is singleton and once it's been created, it will be injectable across the app's life time.

Unlike componet, directive and pipe, if a service needs other dependencies, the service needs @Injectable decorator. Refer to ng:http:get

Angular services :: Http, FormBuilder, Router

  1. FormBuilder ng:FormBuilder
    • Saves from importing FormControl and FormGroup
    import { Component } from '@angular/core';
import { Validators, FormBuilder } from '@angular/forms';

@Component({
  selector: 'mw-media-item-form',
  templateUrl: 'app/media-item-form.component.html',
  styleUrls: ['app/media-item-form.component.css']
})
export class MediaItemFormComponent {
  form;

  constructor(private formBuilder: FormBuilder) {}

  ngOnInit() {
    this.form = this.formBuilder.group({
  medium: this.formBuilder.control('Movies'),
  name: this.formBuilder.control('', Validators.compose([
    Validators.required,
    Validators.pattern('[\\w\\-\\s\\/]+')
  ])),
  category: this.formBuilder.control(''),
  year: this.formBuilder.control('', this.yearValidator),
    });
  }

  yearValidator(control) {
    if (control.value.trim().length === 0) {
  return null;
    }
    let year = parseInt(control.value);
    let minYear = 1800;
    let maxYear = 2500;
    if (year >= minYear && year <= maxYear) {
  return null;
    } else {
  return {
    'year': {
      min: minYear,
      max: maxYear
    }
  };
    }
  }

  onSubmit(mediaItem) {
    console.log(mediaItem);
  }
}
  2. Custom service and inject variable

    https://github.com/coursefiles/angular2-essential-training/tree/05_06b/app https://github.com/coursefiles/angular2-essential-training/tree/05_07b/app

    app.module.ts 

    import { MediaItemService } from './media-item.service'; // inject as a class

// inject as a value
// https://angular.io/api/core/ValueProvider
const lookupLists = {
  mediums: ['Movies', 'Series']
};

@NgModule({
  imports: [ ... ],
  declarations: [ ... ],
  providers: [
    MediaItemService,
    { provide: 'lookupListToken', useValue: lookupLists }
  // Another value provider
  // , { provide: 'lookupListToken2', useValue: 'abc' }
  ],
  bootstrap: [
    AppComponent
  ]
})
export class AppModule {}

    media-item.service.ts 

    export class MediaItemService {
  get() {
    return this.mediaItems;
  }

  add(mediaItem) {
    this.mediaItems.push(mediaItem);
  }

  delete(mediaItem) {
    let index = this.mediaItems.indexOf(mediaItem);
    if(index >= 0) {
      this.mediaItems.splice(index, 1);
    }
  }

  mediaItems = [
    {
      id: 1,
      name: "Firebug",
      medium: "Series",
      category: "Science Fiction",
      year: 2010,
      watchedOn: 1294166565384,
      isFavorite: false
    },
    {...}
  ];
}

    media-item-list.component.ts 

    import { Component } from '@angular/core';

import { MediaItemService } from './media-item.service';

@Component({
  selector: 'mw-media-item-list',
  templateUrl: 'app/media-item-list.component.html',
  styleUrls: ['app/media-item-list.component.css']
})
export class MediaItemListComponent {
  mediaItems;

  constructor(private mediaItemService: MediaItemService) {}

  ngOnInit() {
    this.mediaItems = this.mediaItemService.get();
  }

  onMediaItemDelete(mediaItem) {
    this.mediaItemService.delete(mediaItem);
  }

}

    media-item-form.component.ts 

    import { Component, Inject } from '@angular/core';
// Inject is for injecting the ValueProvider

import { Validators, FormBuilder } from '@angular/forms';

import { MediaItemService } from './media-item.service';

@Component({
  selector: 'mw-media-item-form',
  templateUrl: 'app/media-item-form.component.html',
  styleUrls: ['app/media-item-form.component.css']
})
export class MediaItemFormComponent {
  form;

  constructor(
    private formBuilder: FormBuilder,
    private mediaItemService: MediaItemService,
    @Inject('lookupListToken') public lookupLists) {}

  // private can be used inside the class here as this.xxx
  // public can be used outside the class such as in template as xxx

  ngOnInit() {
    this.form = this.formBuilder.group({ ... });
  }

  yearValidator(control) { ... }

  onSubmit(mediaItem) {
    this.mediaItemService.add(mediaItem);
  }
}

    media-item-form.component.html 

    <form
  [formGroup]="form"
  (ngSubmit)="onSubmit(form.value)">
  <ul>
    <li>
      <label for="medium">Medium</label>
      <select name="medium" id="medium" formControlName="medium">
        <option *ngFor="let medium of lookupLists.mediums" value="{{medium}}">{{medium}}</option>
      </select>
    </li>
    ...li
  </ul>
</form>
  3. OpaqueToken, InjectionToken

    Use this rather than ValueProvider OpaqueToken is deprecated, use InjectionToken https://angular.io/guide/dependency-injection#injection-token

    https://github.com/coursefiles/angular2-essential-training/tree/06_01b/app

    providers.ts 

    import { OpaqueToken } from '@angular/core';

export const lookupListToken = new OpaqueToken('lookupListToken');

export const lookupLists = {
  mediums: ['Movies', 'Series']
};

    app.module.ts 

    import { lookupListToken, lookupLists } from './providers';

@NgModule({
  imports: [ ...  ],
  declarations: [ ... ],
  providers: [
    MediaItemService,
    { provide: lookupListToken, useValue: lookupLists }
  ],
  bootstrap: [
    AppComponent
  ]
})
export class AppModule {}

    media-item-form.component.ts 

    import { lookupListToken } from './provders';

 constructor(
    private formBuilder: FormBuilder,
    private mediaItemService: MediaItemService,
    @Inject(lookupListToken) public lookupLists) {}
  4. Http

    https://angular.io/guide/http

    1. mock back end ng:mock back end

      https://angular.io/guide/http#in-mem-web-api https://angular.io/api/http/XHRBackend https://github.com/coursefiles/angular2-essential-training/tree/06_03b/app

      app.module.ts 

      import { NgModule } from '@angular/core';
import { BrowserModule } from '@angular/platform-browser';
import { ReactiveFormsModule } from '@angular/forms';
import { HttpModule, XHRBackend } from '@angular/http';

import { AppComponent } from './app.component';
import { MediaItemComponent } from './media-item.component';
import { MediaItemListComponent } from './media-item-list.component';
import { FavoriteDirective } from './favorite.directive';
import { CategoryListPipe } from './category-list.pipe';
import { MediaItemFormComponent } from './media-item-form.component';
import { MediaItemService } from './media-item.service';
import { lookupListToken, lookupLists } from './providers';
// new
import { MockXHRBackend } from './mock-xhr-backend';

@NgModule({
  imports: [
    BrowserModule,
    ReactiveFormsModule,
    HttpModule
  ],
  declarations: [
    AppComponent,
    MediaItemComponent,
    MediaItemListComponent,
    FavoriteDirective,
    CategoryListPipe,
    MediaItemFormComponent
  ],
  providers: [
    MediaItemService,
    { provide: lookupListToken, useValue: lookupLists },
    // new
    { provide: XHRBackend, useClass: MockXHRBackend }
  ],
  bootstrap: [
    AppComponent
  ]
})
export class AppModule {}

      https://github.com/coursefiles/angular2-essential-training/blob/06_03b/app/mock-xhr-backend.ts mock-xhr-backend.ts

    2. get ng:http:get

      http.get() returns an Observable<Response>. Refer to js:observable Observable.map() is one of RxJS operators needed to be imported

      RxJS is a 3rd party library implements async Observable pattern.

      https://github.com/coursefiles/angular2-essential-training/tree/06_04b/app

      It uses the paths defined in ng:mock back end

      media-item.service.ts 

      // new
import { Injectable } from '@angular/core';
import { Http } from '@angular/http';
import 'rxjs/add/operator/map';

// This custom service class needs dependency injection, so it needs a decorator Injectable
@Injectable()
export class MediaItemService {
  constructor(private http: Http) {}

  get() {
    // new
    return this.http.get('mediaitems')
      .map(response => {
        return response.json().mediaItems;
      });
  }

  add(mediaItem) {
    this.mediaItems.push(mediaItem);
  }

  delete(mediaItem) {
    let index = this.mediaItems.indexOf(mediaItem);
    if(index >= 0) {
      this.mediaItems.splice(index, 1);
    }
  }

  mediaItems = [ {}, ... ];
}

      media-item-list.component.ts 

      ngOnInit() {
    this.getMediaItems(this.medium);
}
getMediaItems(medium) {
    this.medium = medium;
    this.mediaItemService.get()
      .subscribe(mediaItems => {
        this.mediaItems = mediaItems;
      });
}
    3. get with URLSearchParams

      https://github.com/coursefiles/angular2-essential-training/tree/06_05b/app

      media-item.service.ts 

      import { Injectable } from '@angular/core';
import { Http, URLSearchParams } from '@angular/http';
import 'rxjs/add/operator/map';
// Other Observable operators
// import 'rxjs/add/operator/catch'

@Injectable()
export class MediaItemService {
  constructor(private http: Http) {}

  get(medium) {
    let searchParams = new URLSearchParams();
    searchParams.append('medium', medium);
    // searchParams.set('medium', medium);
    return this.http.get('mediaitems', { search: searchParams })
      .map(response => {
        return response.json().mediaItems;
      });
  }

  add(mediaItem) { ... }

  delete(mediaItem) { ... }

  mediaItems = [ {}, ... ];
}

      media-item-list.component.ts 

      ngOnInit() {
    this.getMediaItems(this.medium);
}
getMediaItems(medium) {
    this.medium = medium;
    this.mediaItemService.get(medium)
      .subscribe(mediaItems => {
        this.mediaItems = mediaItems;
      });
}
    4. post, put, delete

      https://github.com/coursefiles/angular2-essential-training/tree/07_01b/app

      media-item.service.ts 

      import { Injectable } from '@angular/core';
import { Http, URLSearchParams } from '@angular/http';
import 'rxjs/add/operator/map';

@Injectable()
export class MediaItemService {
  constructor(private http: Http) {}

  get(medium) {
    let searchParams = new URLSearchParams();
    searchParams.append('medium', medium);
    return this.http.get('mediaitems', { search: searchParams })
      .map(response => {
        return response.json().mediaItems;
      });
  }

  add(mediaItem) {
    // new
    return this.http.post('mediaitems', mediaItem)
      .map(response => {});
  }

  delete(mediaItem) {
    // new
    return this.http.delete(`mediaitems/${mediaItem.id}`)
      .map(response => {});
  }
}

      media-item-form.component.ts 

      import { Component, Inject } from '@angular/core';
import { Validators, FormBuilder } from '@angular/forms';

import { MediaItemService } from './media-item.service';
import { lookupListToken } from './providers';

@Component({
  selector: 'mw-media-item-form',
  templateUrl: 'app/media-item-form.component.html',
  styleUrls: ['app/media-item-form.component.css']
})
export class MediaItemFormComponent {
  form;

  constructor(
    private formBuilder: FormBuilder,
    private mediaItemService: MediaItemService,
    @Inject(lookupListToken) public lookupLists) {}

  ngOnInit() {
    this.form = this.formBuilder.group({
      medium: this.formBuilder.control('Movies'),
      name: this.formBuilder.control('', Validators.compose([
        Validators.required,
        Validators.pattern('[\\w\\-\\s\\/]+')
      ])),
      category: this.formBuilder.control(''),
      year: this.formBuilder.control('', this.yearValidator),
    });
  }

  yearValidator(control) {
    if (control.value.trim().length === 0) {
      return null;
    }
    let year = parseInt(control.value);
    let minYear = 1800;
    let maxYear = 2500;
    if (year >= minYear && year <= maxYear) {
      return null;
    } else {
      return {
        'year': {
          min: minYear,
          max: maxYear
        }
      };
    }
  }

  onSubmit(mediaItem) {
    // new
    this.mediaItemService.add(mediaItem)
      .subscribe();
  }
}

      media-item-list.component.ts 

      import { Component } from '@angular/core';

import { MediaItemService } from './media-item.service';

@Component({
  selector: 'mw-media-item-list',
  templateUrl: 'app/media-item-list.component.html',
  styleUrls: ['app/media-item-list.component.css']
})
export class MediaItemListComponent {
  medium = '';
  mediaItems = [];

  constructor(private mediaItemService: MediaItemService) {}

  ngOnInit() {
    this.getMediaItems(this.medium);
  }

  onMediaItemDelete(mediaItem) {
    // new
    this.mediaItemService.delete(mediaItem)
      .subscribe(() => {
        this.getMediaItems(this.medium);
      });
  }

  getMediaItems(medium) {
    this.medium = medium;
    this.mediaItemService.get(medium)
      .subscribe(mediaItems => {
        this.mediaItems = mediaItems;
      });
  }
}
9.14.3.18. Route
  1. base href, config and register routes

    https://github.com/coursefiles/angular2-essential-training/tree/07_03b/app

    index.html 

    <base href="/">

    app.routing.ts 

    import { Routes, RouterModule } from '@angular/router';

import { MediaItemFormComponent } from './media-item-form.component';
import { MediaItemListComponent } from './media-item-list.component';

const appRoutes: Routes = [
  { path: 'add', component: MediaItemFormComponent },
  { path: ':medium', component: MediaItemListComponent },
  { path: '', pathMatch: 'full', redirectTo: 'all' }
];

export const routing = RouterModule.forRoot(appRoutes);

    app.module.ts 

    import { routing } from './app.routing';

@NgModule({
  imports: [
    BrowserModule,
    ReactiveFormsModule,
    HttpModule,
    routing
  ],
  ...
})
  2. router-outlet

    app.component.html Before, both form and list are loaded. 

    <section>
  <header>
    <h1>Media Watch List</h1>
    <p class="description">Keeping track of the media I want to watch.</p>
  </header>
  <mw-media-item-form></mw-media-item-form>
  <mw-media-item-list></mw-media-item-list>
</section>

    After, form or list is loaded based on the current router state. The loaded component is the next sibling of the router-outlet HTML element. 

    <section>
  <header>
    <h1>Media Watch List</h1>
    <p class="description">Keeping track of the media I want to watch.</p>
  </header>
  <router-outlet></router-outlet>
</section>

    https://angular.io/api/router/RouterOutlet

  3. routerLink

    app.component.html 

    <nav>
  <a routerLink="/">
    <img src="media/04.png" class="icon" />
  </a>
  <a routerLink="/Movies">
    <img src="media/03.png" class="icon" />
  </a>
  <a routerLink="/Series">
    <img src="media/02.png" class="icon" />
  </a>
</nav>
  4. read route in component

    media-item-list.component.ts 

    import { Component } from '@angular/core';
import { ActivatedRoute } from '@angular/router'; // new

import { MediaItemService } from './media-item.service';

@Component({
  selector: 'mw-media-item-list',
  templateUrl: 'app/media-item-list.component.html',
  styleUrls: ['app/media-item-list.component.css']
})
export class MediaItemListComponent {
  medium = '';
  mediaItems = [];
  paramsSubscription; // new

  constructor(
    private mediaItemService: MediaItemService,
    private activatedRoute: ActivatedRoute // new
  ) {}

  ngOnInit() {
    // before 
    // this.getMediaItems(this.medium);

    // now
    this.paramsSubscription = this.activatedRoute.params
      .subscribe(params => {
        let medium = params['medium'];
        if(medium.toLowerCase() === 'all') {
          medium = '';
        }
        this.getMediaItems(medium);
      });
  }

  // new
  ngOnDestroy() {
    this.paramsSubscription.unsubscribe();
  }

  onMediaItemDelete(mediaItem) {
    this.mediaItemService.delete(mediaItem)
      .subscribe(() => {
        this.getMediaItems(this.medium);
      });
  }

  getMediaItems(medium) {
    this.medium = medium;
    this.mediaItemService.get(medium)
      .subscribe(mediaItems => {
        this.mediaItems = mediaItems;
      });
  }
}
  5. Navigate in code

    Form submit and navigate in code media-item-form.component.ts 

    import { Component, Inject } from '@angular/core';
import { Validators, FormBuilder } from '@angular/forms';
import { Router } from '@angular/router'; // new

import { MediaItemService } from './media-item.service';
import { lookupListToken } from './providers';

@Component({
  selector: 'mw-media-item-form',
  templateUrl: 'app/media-item-form.component.html',
  styleUrls: ['app/media-item-form.component.css']
})
export class MediaItemFormComponent {
  form;

  constructor(
    private formBuilder: FormBuilder,
    private mediaItemService: MediaItemService,
    @Inject(lookupListToken) public lookupLists,
    private router: Router) {}

  ngOnInit() {
    this.form = this.formBuilder.group({
      medium: this.formBuilder.control('Movies'),
      name: this.formBuilder.control('', Validators.compose([
        Validators.required,
        Validators.pattern('[\\w\\-\\s\\/]+')
      ])),
      category: this.formBuilder.control(''),
      year: this.formBuilder.control('', this.yearValidator),
    });
  }

  yearValidator(control) {
    if (control.value.trim().length === 0) {
      return null;
    }
    let year = parseInt(control.value);
    let minYear = 1800;
    let maxYear = 2500;
    if (year >= minYear && year <= maxYear) {
      return null;
    } else {
      return {
        'year': {
          min: minYear,
          max: maxYear
        }
      };
    }
  }

  onSubmit(mediaItem) {
    this.mediaItemService.add(mediaItem)
      .subscribe(() => {
        // new
        this.router.navigate(['/', mediaItem.medium]);
      });
  }
}
9.14.3.20. Style Guide
  1. File Structure ng:file structure

    https://angular.io/guide/styleguide#file-tree

    src/app/heroes/hero.component.ts 

    @Component({
  selector: 'toh-hero'
})
export class HeroComponent {}

    src/app/heroes/hero-list/hero-list.component.ts 

    import { Component, OnInit } from '@angular/core';

import { Hero, HeroService } from '../shared';

@Component({
  selector: 'toh-heroes',
  template: `
      <pre>{{heroes | json}}</pre>
    `
})
export class HeroListComponent implements OnInit {
  heroes: Hero[] = [];

  constructor(private heroService: HeroService) { }

  ngOnInit() {
    this.heroService.getHeroes().subscribe(heroes => this.heroes = heroes);
  }
}

    src/app/heroes/shared/hero.service.ts|spec.ts 

    import { Injectable } from '@angular/core';
import { Http }       from '@angular/http';

import { Hero } from './hero.model';
import { ExceptionService, SpinnerService, ToastService } from '../../core';
export class HeroService {
  constructor(private http: Http) { }

  getHeroes() {
    return this.http.get('api/heroes')
      .map((response: Response) => <Hero[]>response.json().data);
  }
}

    src/app/heroes/shared/hero.model.ts 

    export class Hero {
  id: number;
  name: string;
}

    src/app/heroes/shared/hero-button.component.ts|html|css|spec.ts

  2. File name

    Use dashes to separate words in file names (hero-list) Use a dot to separate the descriptive name from the type (.component) app/heroes/hero-list.component.ts

    A file should have no more than 400 lines of code. A small function should have no more than 75 lines.

  3. Symbol name and file name

    Use upper camel case for class names 

    // hero-list.component.ts
@Component({ ... })
export class HeroListComponent { }

// validation.directive.ts
@Directive({ ... })
export class ValidationDirective { }

// app.module.ts
@NgModule({ ... })
export class AppModule

// init-caps.pipe.ts
@Pipe({ name: 'initCaps' })
export class InitCapsPipe implements PipeTransform {}

// user-profile.service.ts
@Injectable()
export class UserProfileService { }
  4. Bootstrapping

    Use main.ts as the file name Include error handling Don't put logic in this file. 

    import { platformBrowserDynamic } from '@angular/platform-browser-dynamic';
import { AppModule }              from './app/app.module';
platformBrowserDynamic().bootstrapModule(AppModule)
  .then(success => console.log(`Bootstrap success`))
  .catch(err => console.error(err));
  5. Directive selectors (lower camel case)

9.15. React.js

Documentation
https://react.dev/
React Lifecycle methods diagram
http://projects.wojtekmaj.pl/react-lifecycle-methods-diagram/
(no term)
netlify.com, deploy your build folder to a web server. e.g. it has redirect http to https

9.15.2. create-react-app

# cd to your new project's parent folder
cd /path/to/myProjects

# new way
npx create-react-app lake-app

cd lake-app
npm start

# build a production inside the "build" folder 
npm run build

# then you can serve the prod code.

# First install "serve" as a static server
sudo npm i -g serve

serve -s build

9.15.3. Vite

# uses package create
npm create vite@latest my-react-app

# or the current directory
npm create vite@latest .

9.15.4. react-bootstrap

  • https://react-bootstrap.github.io/getting-started/introduction/
  • react-boostrap itself does not have .css files
  • import 'bootstrap/dist/css/bootstrap.min.css';
    • Or include <link rel="stylesheet" href="..." /> in public/index.html

  • In each of your component, import the required BS component 

    import Jumbotron from 'react-bootstrap/Jumbotron';

// or
import { Button, Jumbotron } from 'react-bootstrap';

9.15.5. react-router-dom

  • https://reacttraining.com/react-router/web/guides/quick-start
  • npm i react-router-dom

  • In src/index.js or App.js 

    import { BrowserRouter as Router, Switch, Route, Link } from 'react-router-dom';

  • Inside your component 

    render () {
    return (
        <Router>
            <header>some header</header>
            <Switch>
                <Route exact path="/" component={Home} />
                <Route path="/vitamin" component={Vitamin} />
            </Switch>
        </Router>
    )
}

9.15.6. JSX

Comment out
{/* and */}
(no term)
Inside (), use {}
(no term)
Inside {}, use ()
9.15.6.1. Ternary return with another JSX
useEffect(() => {
    count && (document.title = `$(pageTitle) - $(count)`)
})

return (
    {this.state.message === 'failed'
     ? (
         <div>
             <row>three elements wrapped</row>
             <row>inside</row>
             <row>another element work.</row>
         </div>
     )
     : (
         <row>html like</row>
         <row>haiku</row>
         <row>must follow rules of structure.</row>
     )
  }
)

9.15.7. TypeScript

React TypeScript Cheatsheet
https://react-typescript-cheatsheet.netlify.app/

9.15.8. CSS

9.15.8.1. Raw CSS file import
// index.js
import './components/MyComponent/MyComponent.css';

// Don't use MyComponent.module.css'

// components/MyComponent/MyComponent.js
// You may import but not necessary because the whole CSS file is imported before in index.js
return (
        <>
           <span className="lili">hello</span>
        </>
)

The final HTML is <span class="lili">hello</span>

9.15.8.2. CSS Module Import
// components/MyComponent/MyComponent.js
import styles from './MyComponent.module.css';

return (
        <>
         <span className={[styles.lili, "lilihello"].join(' ')}>hello</span>
        </>
)

The final HTML is <span class="MyComponent_lili__J4Vn2 lilihello">hello</span> The naming convention is [fileName]_[cssClassName]__[hash]

9.15.8.3. Styled-Component (CSS in JS)
9.15.8.4. NPM classnames
  • https://www.npmjs.com/package/classnames#usage-with-reactjs

  • Compare 

    class Button extends React.Component {
  // ...
  render () {
    var btnClass = 'btn';
    if (this.state.isPressed) btnClass += ' btn-pressed';
    else if (this.state.isHovered) btnClass += ' btn-over';
    return <button className={btnClass}>{this.props.label}</button>;
  }
}

// Using NPM classnames

var classNames = require('classnames');

class Button extends React.Component {
  // ...
  render () {
    var btnClass = classNames({
      btn: true,
      'btn-pressed': this.state.isPressed,
      'btn-over': !this.state.isPressed && this.state.isHovered
    });
    return <button className={btnClass}>{this.props.label}</button>;
  }
}
9.15.8.5. NPM styled-components

9.15.9. Redux

  • Redux Toolkit
  • react-redux
    • A complimentary package which deals with binding
    • npm i -S react-redux

9.15.10. Custom hooks

9.15.10.1. useState
Group states and state setters in a file
usePasswordToggler
// ./src/hooks/usePasswordToggler.js
import {useState} from 'react';

export const usePasswordToggler = () => {
    const [passwordVisibility, setPasswordVisibility] = useState(true);
    const [type, setType] = useState('password');
    const handlePasswordVisibility = () => {
        if (type === 'password') {
            setType('text');
            setPasswordVisibility(!passwordVisibility);
        } else if (type === 'text') {
            setType('password');
            setPasswordVisibility(!passwordVisibility);
        }
    };
    return {
        type,
        passwordVisibility,
        handlePasswordVisibility
    };
};

import React from 'react';
import {usePasswordToggler} from './hooks/usePasswordToggler';
import './App.css';

function App() {
    const {type, passwordVisibility, handlePasswordVisibility} = usePasswordToggler()
    return (
        <main>
            <div>
                <input type={type} placeholder='Enter password...' />
                <button onClick={handlePasswordVisibility}>{passwordVisibility ? 'Show' : 'Hide'} Password</button>
            </div>
        </main>
    );
}
9.15.10.2. useReducer
  1. Sample

    Old code 

    import { useState } from 'react';
import AddTask from './AddTask.js';
import TaskList from './TaskList.js';

let nextId = 3;
const initialTasks = [
    {id: 0, text: 'Visit Kafka Museum', done: true},
    {id: 1, text: 'Watch a puppet show', done: false},
    {id: 2, text: 'Lennon Wall pic', done: false},
];

export default function TaskApp() {
    const [tasks, setTasks] = useState(initialTasks);

    function handleAddTask(text) {
        setTasks([
            ...tasks,
            {
                id: nextId++,
                text: text,
                done: false,
            },
        ]);
    }

    function handleChangeTask(task) {
    }

    function handleDeleteTask(taskId) {
    }

    return (
        <>
            <AddTask onAddTask={handleAddTask} />
            <TaskList
                tasks={tasks}
                onChangeTask={handleChangeTask}
                onDeleteTask={handleDeleteTask} />
        </>
    )
}
    Step 1

    Move from setting state to dispatching actions 

    function handleAddTask(text) {
  dispatch({
    type: 'added',
    id: nextId++,
    text: text,
  });
}
    Step 2

    Write a reducer function which groups all dispatching events that change the state 

    function tasksReducer(tasks, action) {
    switch (action.type) {
        case 'added': {
            return [
                ...tasks,
                {
                    id: action.id,
                    text: action.text,
                    done: false,
                },
            ];
        }
        case 'changed': {
            return '';
        }
        default: {
            throw Error('Unknown action: ' + action.type);
        }
    }
}
    (no term)

    Step 3: Use the reducer 

    import { useReducer } from 'react';
import AddTask from './AddTask.js';
import TaskList from './TaskList.js';

let nextId = 3;
const initialTasks = [
    {id: 0, text: 'Visit Kafka Museum', done: true},
    {id: 1, text: 'Watch a puppet show', done: false},
    {id: 2, text: 'Lennon Wall pic', done: false},
];

export default function TaskApp() {
    const [tasks, dispatch] = useReducer(tasksReducer, initialTasks);

    function handleAddTask(text) {
        dispatch({
            type: 'added',
            id: nextId++,
            text: text,
        });
    }

    function handleChangeTask(task) {}

    function handleDeleteTask(taskId) {}

    return (
        <>
            <h1>Prague itinerary</h1>
            <AddTask onAddTask={handleAddTask} />
            <TaskList
                tasks={tasks}
                onChangeTask={handleChangeTask}
                onDeleteTask={handleDeleteTask}
            />
        </>
    );
}
9.15.10.3. useContext
  • Use context to pass properties from parent component to distant children
  • https://react.dev/learn/passing-data-deeply-with-context
  • LevelContext.js
    • createContext to initiate context
  • <LevelContext.Provider value={initialLevel}>
    • In parent component
    • If any component inside this parent component asks for LevelContext, give them this initialLevel. The component will use the value of the nearest <LevelContext.Provider> in the UI tree above it.

9.15.11. Use absolute imports

By default, React apps with nested directory structures might lead to confusion on the import path such as the example below: 

import { Button } from '../../components';

Specify base path in jsconfig.json: 

{ 
    "compilerOptions": { 
        "baseUrl": "src" 
    }, 
    "include": ["src"]
}

import { Button } from 'components';

React picks up jsconfig.json or tsconfig.json to resolve path in config/paths.js

If your project includes webpack.config.js at the root, you may further personalize it by using a prefix such as @components or ~components. See the configuration below: 

module.exports = {
    resolve: {
        extensions: ['js'],
        alias: {
            '@': path.resolve(__dirname, 'src'),
            '@components': path.resolve(__dirname, 'src/components')
            '@hooks': path.resolve(__dirname, 'src/hooks'),
        }
    }
};

import { Button } from '@components';

9.15.13. Incrementally Migrate to React

  • Opptunities of migration
    • Faster browser rendering speed (open loan > close loan > reopen loan)
      • PHP returns data only instead of bytes of HTML
      • React only renders the parts of HTML that are changed
        • e.g. a list of borrowers under loan view, remove a borrower from loan, PHP returns the whole Borrower section again. Instead, PHP returns the updated data of the loan's borrowers, and React will remove that removed Borrower div. Less amount of data is transferred, less re-rendering
    • Front end and Back end decoupling
      • Reduce risk
        • Remove a lot of spaghetti PHP code that was mixed with HTML
      • Provide an opportunity for us to rewrite the PHP code and standardize the data
      • Ship new frontend features faster
  • Risk of migration to React
    • Risk in rewriting PHP code to output/supply data
      • One challenge is the action rewrite, e.g. JavaScript and PHP delloanclient(clientId, loanId) returns HTML of borrowers of a loan, now we need to return data and then update the state.
      • A/B testing, Laravel Facade concept, Adaptor Pattern, delloanclientv1, delloav2, delloanclient
  • Metrics
    • Loan view time to interactive
    • Size of the app
      • memory usage
      • state memory
      • number of event handlers
  • Step by Step
    • Choose small basic components first
      • Basic UI components like buttons
      • Lowest hanging fruits
    • New feature has to have front end and back end decoupled
      • What's the data we should provide?
      • New feature needs basic UI components. Develop those.
  • Data
    • Actions of Data
      • CRUD
      • Form
    • In other words, data first

9.15.14. React State data structure

  • gss
    • country
    • almodules
  • loans
    • loans table row in almost full detail (hide not-safe-for-public fields)
      • loanid: 123
        borrowers
        not the full borrower detail
        • id:
        • order:
  • borrowers
    • borrower in full detail
      client_id
      756
      first_name
      last_name
      corporation_name

9.15.15. Directory Structure

  • src
    • assets
    • components
      Input
      a component called Input
      Input.js
      all the logic
      (no term)
      Input.css
      (no term)
      Input.test.js
    • pages
      • Strictly views, no business logic
      • They are also components
    • codes that interact with external API resources
    • utils
    • hooks
    • store

9.17. CryptoJS Goolge Code Archive

10. NodeJS

10.1. Install

10.1.1. Ubuntu

apt-get update

# 18.04 has the first installed but not the libssl-dev
apt-get install build-essential libssl-dev

# Just go to https://github.com/nvm-sh/nvm to run the install script
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash

source ~/.bashrc
  • Directory ~/.nvm is created with the cloned code
  • Insert something to the corresponding profile file
    • ~/.bashrc or bash_profile or .profile or .zshrc
  • Log out and back again or source the file so that current session knows changes
  • List Node.js available versions nvm ls-remote and look for version with Latest LTS
# nvm install 10.14.1
# nvm use 10.14.1

nvm install v16.14.0

# What versions are installed
nvm ls

# Current in-use node version
node -v

# Make one version as default for new sessions
nvm alias default 6.10.2

# Use the default version
nvm use default

# to uninstall, first make sure the target version is not the current active version
nvm current
nvm uninstall node_version

# if the target version is the current active version, first deactivate
nvm deactivate
nvm uninstall node_version

Global packages installed using npm install -g express are in ~/.nvm/versions/node/node_version/lib/node_modules/express

https://github.com/nodejs/LTS

10.1.1.1. Install using PPA
curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
sudo apt-get install -y nodejs

# you might need to install gnupg
sudo apt-get install -y gnupg

# might need to install npm if nodejs doesn't have it
sudo apt-get install -y npm

nodejs -v
npm -v

# install build-essential so that npm can compile code from source
sudo apt install build-essential

10.1.2. MacOS

brew update
# other alias: nodejs
# will install the latest version (not the LTS version)
brew install node
# node, npm and npx are installed
node -v
npm -v
npx -v

10.1.3. Install on Windows

Download the .msi and install with NPM. Installation path: C:\Program Files\nodejs\ Download the latest .msi to upgrade. And upgrade any existing global packages.

Upgrade npm on Windows

  • Just use the latest .msi to upgrade is good enough to upgrade npm for Windows
  • https://www.npmjs.com/package/npm-windows-upgrade
  • PowerShell with admin Set-ExecutionPolicy Unrestricted -Scope CurrentUser -Force
  • npm install --global --production npm-windows-upgrade

Refer to phpstorm:nodejs

10.2. Basic

  • Modules installed globally usually have an executable command for direct use.
    • For locally installed modules, usually run ./node_modules/module_name/bin/exec_cmd
  • https://node.green/
    Find V8 versio
    node -p process.versions.v8 shows V8.ver.number-node.nodeVersionNumber
    More
    https://nodejs.org/en/docs/es6/
node -v
npm -v

# clear cache
npm cache clean --force

# or
npm cache verify

# Install npm package to ./node_modules
npm install express

# install without arguments will install dependencies including devDependencies
npm install

# with flag or when NODE_ENV is set to production, it will not install devDependencies
npm install --production

# with --only={prod[uction]|dev[elopment]} to install only non-devDependencies or devDependencies
npm install --only=dev

# Install package globally (available to other projects using the same Node.js version)
# ~/.nvm/node_version/lib/node_modules/package_name
npm install -g gulp

# What packages/modules are installed globally, just remove -g for locally
npm ls -g

# Check which global packages are outdated
npm outdated -g

# Update all global packages. Some global package update requires removing first
npm update -g

# Update one global package
npm install -g <packagename>

# Remove a package globally, just remove -g for locally
npm remove async -g

Link 

# run npm link alone in local repo is to create a symlink in global folder that links to the package where the command was executed
cd ~/projects/node-redis # Note local package name is not node-redis but redis as in ~/projects/node-redis/package.json
npm link # create symlink in global folder so that redis can be globally referrenced/linked.

cd ~/projects/node-bloggy # go to another project
npm link redis # install redis package in ~/projects/node-bloggy/node_modules/node-redis, file changes in ~/projects/node-redis/ will be reflected.

npm link express # a global package can be linked inside any local repo

10.3. Global Object

console.log is actually global.console.log https://nodejs.org/api/globals.html

10.3.1. __dirname, __filename

Full path of the directory/file of the current module/file

10.3.2. console nodejs:console

Backtick 

var a = "World";
console.log(`Hello ${a}`); // Hello World

// console.dir(obj[, options]) uses util.inspect()
console.dir(a, {
    showHidden: true,
    depth: 2, // number, can be null
    colors: false
});

10.3.3. require

10.3.3.1. CommonJS
// Importing a local module with a path relative to the `__dirname` or current
// working directory. (On Windows, this would resolve to .\path\myLocalModule.)
const myLocalModule = require('./path/myLocalModule');

// Importing a JSON file:
const jsonData = require('./path/filename.json');

// Importing a module from node_modules or Node.js built-in module:
const crypto = require('crypto');
10.3.3.2. Asynchronous Module Definition (AMD)
  • Features
    • Asynchronous and made for frontend. Exact opposite of CJS

  • Code example 

    // define is a RequireJS function

// Define one module
define(['dependency1', 'dependency2'], function(dependency1, dependency2) {
  // Methods
  function myPublicMethod() {
    console.log('Hello, world!');
  }

  // Exposed public methods
  return {
    myPublicMethod: myPublicMethod
  };
});
10.3.3.3. Universal Module Definition (UMD)
  • Features
    • For apps that work on both synchronous and asynchronous environment. Browser and Server apps.
    • It's more like a pattern to configure several module systems

  • Code example 

    (function (root, factory) {
  if (typeof define === 'function' && define.amd) {
    // AMD. Register as an anonymous module.
    define(['dependency'], factory);
  } else if (typeof module === 'object' && module.exports) {
    // Node. Does not work with strict CommonJS, but
    // only CommonJS-like environments that support module.exports,
    // like Node.
    module.exports = factory(require('dependency'));
  } else {
    // Browser globals (root is window)
    root.returnExports = factory(root.dependency);
  }
}(this, function (dependency) {
  // Methods
  function myModule() {
    console.log('Hello, world!');
  }

  // Exposed public methods
  return {
    myModule: myModule
  };
}));
10.3.3.4. ECMAScript Module (ESM)
  1. Code example
    // https://stackoverflow.com/questions/39282253/how-can-i-alias-a-default-import-in-javascript

import {default as myAlias} from 'my-module';
// eq. to
import myAlias from 'my-module';

// React is the react.default and useEffect is one of the exports
import React, {useEffect} from 'react';

// e.g. MyReact.ReactElement, but not MyReact.default
import * as MyReact from 'react';

// a more complicated example
// material/Chip/index.js
export { default } from './Chip'; // Chip.js, default is const Chip. Then index.js exported default is const Chip
export { default as chipClasses } from './chipClasses';
export * from './chipClasses';

// export default using arrow function
export default () => {
}

// eq.
export default function MyComponent() {
}

export function MyComponent() {
}

// eq. (I don't like it)
export const MyComponent = () => {
}

// MyComponent/MyComponent.js
import Chip from '../material/Chip';

    Import from ES5 

    // tailwindcss/forms
module.exports = forms

import whateverNameYouWant from '@tailwindcss/forms'

10.3.4. process nodejs:process

  • console.log(process.pid)
    • process.versions.node
10.3.4.1. process.argv
node app.js --user li --greeting "Good Day Sir"

app.js 

console.log(process.argv);
// ['nodejs installation full path', 'full path to module', '--user', 'li', '--greeting', 'Good Day Sir']

const grab = flag => {
    let index = process.argv.indexOf(flag)+1;
    return process.argv[index];
}

var greeting = grab('--greeting');
10.3.4.2. process.stdout, process.stdin, process.exit(), process.on()
var q = ['Q1', 'Q2'];
var a = [];
function ask(i) {
  process.stdout.write(`\n\n ${q[i]}`);
  process.stdout.write(' > ');
}

process.stdin.on('data', function(data) {
  answers.push(data.toString().trim());
  if (a.length < q.length) {
    ask(a.length);
  }
  else {
    process.exit();
  }
});

process.on('exit', function() {
  process.stdout.write(`${a[0]}, ${a[1]}, ${a[2]}`);
});

ask(0);
10.3.4.3. process.stdout.clearLine(), process.stdout.cursorTo(), process.stdout.write()
var currentTime = 0;
var percentWaited = 0;

function writePercent(p) {
  process.stdout.clearLine();
  process.stdout.cursorTo(0);
  process.stdout.write(`waiting ... ${p}%`);
}

var interval = setInterval(functino() {
  currentTime += 500;
  percentWaited = Math.floor((currentTime/3000) * 100);
  writePercent(percentWaited);
}, 500);

setTimeout(function() {
  clearInterval(interval);
  writePercent(100);
  console.log("\n\n done \n\n");
}, 3000);

writePercent(percentWaited);

10.3.5. module

./lib/Person.js 

// some code
module.exports = Person; // literal object or any Javascript type

// even export an anonymous function will work

// module.exports is the object that is returned by the require statement

./app.js 

var Person = require('./lib/Person'); // do not include .js
var ben = new Person('Ben');
var george = new Person('George');

10.4. Modules, Packages, Frameworks

10.4.1. Get version

npm list
locally installed packages
npm list -g
globally installed packages
npm list -g | grep gulp
to see if a package is installed globally
npm list -g --depth=0
global installed packages at first level
(no term)
npm list express
npm ls -depth 0
including devDependencies
npm ls -dev
devDependencies only
npm ls -dev -depth 0
list first level devDependencies
npm view PKG_NAME dependencies
list dependencies of a package
npm view PKG_NAME versions --json
The latest available version of a package
view aliaes
info, show, v
Distribution tags
npm view mjml dist-tags
latest
no pre-release/beta versions
beta
optional
future
optional
next
usually is set to a beta version to be the next version
(no term)
Backup and remove the node_modules and run npm install mjml@next
Update npm
sudo npm i -g npm
(no term)
Install packages
npm i -D
eq. to npm install --save-dev gulp-imagemin
  • devDependencies are the ones only needed for development not in production or testing environments
(no term)
By default, the dependency is saved since v5~npm i -S~
Legacy
npm i -S gulp-imagemin eq. npm install --save gulp-imagemin
Npm website for a package gulp-sourcemaps
https://www.npmjs.com/package/gulp-sourcemaps

10.4.2. Semver ~ vs ^

^1.2.3
any 1.x.x latest release but will miss 2.0.0 (e.g. allow minor and patch but not major)
~1.2.3
any 1.2.x versions but will miss 1.3.0 (e.g. allow patch but not minor or major)
^
caret, is the default now

10.4.4. Update Package

npm outdated -l
Show if there're updates for locally installed packages based on package.json in --long format
  • package status
    • Current
    • it may be higher than Latest as it's next in dist-tags above
    • same as latest in dist-tags above
  • A package is outdated when its Current is lower than Latest. It'll update to Latest. If Current is higher than Latest, then it'll be downgraded. package.json also limits what versions can be updated to so don't be surprised after npm update there're still records in npm outdated
npm outdated --global
for globally installed packages
npm outdated -g --depth=0
see which top level globally installed packages need to be updated
npm update -g <package_name>
update a single global package
npm update -g
updaste all global packages
  • Some packages require administrative privilege. On Windows open PowerShell as Admin
npm update
update all outdated "dependencies" in package.json and change package.json to save the new version as the minimum required dependency.
npm update --dev
updates all outdated "dependencies" and "devDependencies" in package.json
npm update sharp
update a local package to the latest
(no term)
General package update process
npm outdated
see which packages have updates. npm outdated -l for long format
  • compare what is installed with package.json
npm update
update packages and package.json
npm audit
check vulnerabilities
npm install
sometimes it's needed..
(no term)
Resolve vulnerabilities after npm audit fix

  • You should rely on first-level contributers to solve the problem. Make sure you update the first-level packages. If you still want to resolve it, cautiously follow these steps:

    • npm audit find the problematic packages from the last part of Path, e.g. package minimist
    • npm ls minimist
    • npm i minimist --save-dev
    • Add this to package.json
    "resolutions": {
    "minimist": "^1.2.5"
}
    npm users need to install this package
    npm i npm-force-resolutions --save-dev, add this to package.json
    "scripts": {
  "preinstall": "npx npm-force-resolutions"
     }
    • Try delete node_modules and package-lock.json
    • npm install, npm update, npm audit fix and npm audit
    • Some vulnerabilties cannot be solved
    • First-level package has hard coded the dependency version number
    • No patch available yet

10.4.5. Scoped package

npm install -g @angular/cli where @ indicates angular/cli is a scoped package. Without @, package is installed in node_modules/packagename Scoped package is installed in node_modules/@angular/cli Thus, scoped package can group packages under one namespace

  • package.json 

    "dependencies": {
  "@angular/cli": "^1.3.0"
}

  • In code 

    require('@angular/cli')

10.4.6. Peer dependencies

A dependency says "I need this thing directly available to me" A peerDependency says "If you want to use me, you need this ting available to you"

However, as of verison 3, npm does not install packages listed in peerDependencies sections.

But npm issues a warning when

  • When any peer dependencies are missing
  • or when the application or any of its other dependencies installs a different version of a peer dependency.

It is your job to copy all peer dependency packages to devDependencies to install them.

For example, you app depends on angular and your app doesn't have peerDependencies. However, angular has peerDependencies.

You will make sure to list any angular's peerDependencies into your app's devDepenedencies

10.4.7. Core modules

10.4.7.1. v8 - core nodejs:v8
var v8 = require('v8');
console.log('v8.getHeapStatistics()');
10.4.7.2. util - core nodejs:util
var EventEmitter = require('events').EventEmitter;
const util = require('util');
var Person = function(name) {
  this.name = name;
};

util.inherits(Person, EventEmitter);
// Add EventEmitter to Person's prototype
// Person inherits all from EventEmitter

var ben = new Person("Ben");

ben.on('speak', function(said) {
  console.log(`${this.name} said ${said}`);
});

ben.emit('speak', "I'm Ben");

// fully print an object
console.log(util.inspect(myObj, {showHidden: false, depth: null}));
// or this for short
cosole.log(util.inspect(myObj, false, null));
// console.dir actually uses util.inspect. See nodejs:console

util.format('%s:%s', 'foo'); // 'foo:%s'
util.format('%s:%s', 'foo', 'bar', 'baz'); 
// extra arguments are coerced into strings delimited by a space 
// 'foo:bar baz'
// %s > string, %d > integer or floating point value
// %i > integer, %f > floating point value
// %j > JSON
// %% > percent sign '%'
10.4.7.3. path - core nodejs:path
var path = require('path');
// path is native, get just the file name
console.log(path.basename(__filename));

// No trailing '/'
var dirUploads = path.join(__dirname, 'www', 'files', 'uploads');
10.4.7.4. readline - core nodejs:readline
  • rl.createInterface(options)
  • rl.question(query, callback)
  • Events
    • close
var readline = require('readline');
var rl = readline.createInterface(process.stdin, process.stdout);
var aPerson = {
  name: '',
  sayings: []
}

rl.question("What is the name of a real person? ", function(a) {
  aPerson.name = a;
  rl.setPrompt(`What would ${aPerson.name} say? `);
  rl.prompt(); // display prompt
  rl.on('line', function(saying) {
    aPerson.sayings.push(saying.trim());
    if (saying.toLowerCase().trim() === 'exit') {
      rl.close();
    }
    else {
      rl.setPrompt(`What else would ${aPerson.name} say?  ('exit' to leave) `);
      rl.prompt();
    }

  });
});

rl.on('close', function() {
  console.log("5s is a real person that says %j", aPerson.name, aPerson.sayings);
  process.exit();
});
10.4.7.5. events - core nodejs:events
e.EventEmitter()
emitter class
  • emitter.on('eventName', callback)
  • emitter.emit('eventName', var1, var2)
var events = require('events');
var emitter = new events.EventEmitter();

emitter.on('customEvent', (msg, status) => {
  console.log(`${status}: ${msg}`);
});

emitter.emit('customEvent', "Hello", 200);
10.4.7.6. fs - core nodejs:fs
  • Default fs is not a Promise although it is async with callback fs.readdir( './abc', callback)
    • They cannot be chained. Consider nodejs:bluebird or nodejs:co-fs
    • Every function has a sync version. Just append Sync at the end
      • let theFiel = fs.readdirSync('./abc')
  • fs.readFile('path', 'utf-8', callback(err, data))
  • fs.readdir('path', callback(err, data))
  • stats
    • stats.isFile()
  • fs.writeFile('path', data)
  • fs.rename
  • fs.unlink
  • fs.appendFile
  • fs.mkdir, fs.rmdir
  • fs.createReadStream, fs.createWriteStream
var fs = require('fs');
var data = require('./data1.json');
console.log(data.name); // data is an object
fs.readFile('./data1.json', 'utf-8', (err, data) => {
  // without specifying utf-8, file will be read as binary
  data = JSON.parse(data); // data is string
  console.log(data.name);
  // res.setHeader('Content-Type', 'text/html');
  // res.send(html);
});

// Read directory
fs.readdir('/root', function(err, data) {
  console.log(data); // data is array
  data.forEach(function(fname) {
    var file = path.join(__dirname, "lib", fname);
    var stats = fs.statSync(file);
    if (stats.isFile() && fname !== '.DS_Store') {
      fs.readFile(file, "UTF-8", function(err, data) {
        console.log(data);
      });
    } 

  });
});

// Write file
var tomString = '{ "name": "Tom" }';
fs.writeFile('tom.json', tomString, function(err) {});
var timmy = {
  name: 'Timmy'
};

fs.writeFile('timmy.json', JSON.stringify(timmy));

// Append file
fs.appendFile('timmy.json', 'more text \n');

// Rename file
fs.rename('./lib/project-config.js', './lib/config.json');

// Move file
fs.rename('./lib/project-config.js', './config.json');

// Remove file
fs.unlink('./lib/config.json');

// New folder
if (!fs.existsSync("lib")) {
  fs.mkdir("lib", function(err) {});
}

// Rename or move
fs.rename('./assets/logs', './logs');

// Remove folder, folder must be empty
fs.rmdir('./logs');

// Remove all files from a directory
fs.readdirSync('./logs').forEach(function(fileName) {
  fs.unlinkSync('./logs/' + fileName);
});

// File read stream
var stream = fs.createReadStream('./chat.log', 'UTF-8');
var data = "";
stream.once('data', function() {
  console.log('Start reading file');
});
stream.on("data", function(chunk) {
  process.stdout.write(` chunk: ${chunk.length} |`);
  data += chunk;
});
stream.on('end', function() {
  console.log(data.length);
});

// File write stream
var stream = fs.createWriteStream('result.md');
stream.write(`${data}\n`);
stream.close();
10.4.7.7. child_process - core nodejs:child_process

Exec is for short process 

var exec = require('child_process').exec;
exec('ls', function(err, stdout) {
  if (err) throw err;
  console.log(stdout);
});

Spawn for ongoing process (async) 

var spawn = require('child_process').spawn;
var cp = spawn('node', ["app"]); // node app
cp.stdout.on("data", function(data) {
  // Output child process stdout
  console.log(`STDOUT: ${data.toString()}`);
});

cp.on("close", function() {
  console.log("Child process has ended.");
  process.exit();
});

setTimeout(function() {
  // maximum run time
  cp.stdin.write("stop");
}, 4000);

10.4.8. npx

  • Run <command> either from a local ./node_modules/.bin/<command>, or from a central cache, installing any packages needed in order for <command> to run
  • By default, it will check whether <command> exists in $PATH, or in the local project binaries, and execute that. If <command> is not found, it will be installed prior to execution
# should already be installed
npm i -g npx

# usage
# npx [options] <command>[@version] [command-arg]...

# npx [options] [-p|--package <pkg>]... <command> [command-arg]...

# npx [options] -c '<command-string>'

# npx --shell-auto-fallback [shell]

10.4.9. BabelJS

Try out
https://babeljs.io/
CDN
https://babeljs.io/docs/babel-standalone
(no term)
Babel vs TypeScript
  • Babel doesn't check for types
  • Babel only compiles one file at a time while TS compiles an entire project
(no term)
Modules
10.4.9.1. Config file
  • Project-wide config
    babel.config.json
    useful when there're multiple package directories e.g. multiple package.json
  • File-relative config
    .babelrc.json
    .babelrc is an alias
    (no term)
    package.json file with a babel key
{
  "presets": [
    "@babel/preset-env"
  ]
}

10.4.10. axios

  • npm i axios
  • imort axios from 'axios'
  • axios.post(url[, data[, config]])
  • Request config

10.4.11. lodash nodejs:lodash

# npm i -g npm
npm i --save lodash

var _ = require('lodash');

var users = [
  { 'user': 'barney',  'age': 36, 'active': true },
  { 'user': 'fred',    'age': 40, 'active': false },
  { 'user': 'pebbles', 'age': 1,  'active': true }
];

_.find(users, function(o) { return o.age < 40; });
// => object for 'barney'

// The `_.matches` shorthand.
_.find(users, { 'age': 1, 'active': true });
// => object for 'pebbles'

// The `_.matchesProperty` shorthand.
_.find(users, ['active', false]);
// => object for 'fred'

// The `_.property` shorthand.
_.find(users, 'active');
// => object for 'barney'

var user = _.findIndex(users, {id: req.params.id});
var update = req.body;
if (update.id) {
  delete update.id;
}

if (!users[user]) {
  res.send('user not found');
}
else {
  var updatedUser = _.assign(users[user], update);
  // merge multiple objects from left to right for enumerable string keyed properties
  res.json(updatedUser);
}
10.4.11.1. _.identity(value)

returns the first argument it receives

var object = { 'a': 1 }; console.log(_.identity(object) = object); // => true

10.4.11.2. _.matches(source)

creates a function that performs a partical deep comparison between a given object and source, return true if the given object has equivalent prop values, else false. Partial comparisons will match empty array and empty object source values against any array or object value, respectively. 

var objects = [
  { 'a': 1, 'b': 2, 'c': 3 },
  { 'a': 4, 'b': 5, 'c': 6 }
];

_.filter(objects, _.matches({ 'a': 4, 'c': 6 }));
// => [{ 'a': 4, 'b': 5, 'c': 6 }]
10.4.11.3. _.find, _.findLast

Return the first element that the predicate function returns true. 

_.find(collection, [predicate=_.identity], [fromIndex=0])
_.findLast(collection, [predicate=_.identity], [fromIndex=collection.length-1])

// Predicate function is passed with (value, index|key, collection)

var users = [
  { 'user': 'barney',  'age': 36, 'active': true },
  { 'user': 'fred',    'age': 40, 'active': false },
  { 'user': 'pebbles', 'age': 1,  'active': true }
];

_.find(users, function(o) { return o.age < 40; });
// => object for 'barney'

// The `_.matches` iteratee shorthand.
_.find(users, { 'age': 1, 'active': true });
// => object for 'pebbles'

// The `_.matchesProperty` iteratee shorthand.
_.find(users, ['active', false]);
// => object for 'fred'

// The `_.property` iteratee shorthand.
_.find(users, 'active');
// => object for 'barney'
10.4.11.4. _.iteratee

Creates a function that invokes func with the arguments of the created function. func is

a property name
the created function returns the prop value for a given element.
_.iteratee([func=_.identity])

var users = [
  { 'user': 'barney', 'age': 36, 'active': true },
  { 'user': 'fred',   'age': 40, 'active': false }
];

// The `_.matches` iteratee shorthand.
_.filter(users, _.iteratee({ 'user': 'barney', 'active': true }));
// => [{ 'user': 'barney', 'age': 36, 'active': true }]

// The `_.matchesProperty` iteratee shorthand.
_.filter(users, _.iteratee(['user', 'fred']));
// => [{ 'user': 'fred', 'age': 40 }]

// The `_.property` iteratee shorthand.
_.map(users, _.iteratee('user'));
// => ['barney', 'fred']

// Create custom iteratee shorthands.
_.iteratee = _.wrap(_.iteratee, function(iteratee, func) {
  return !_.isRegExp(func) ? iteratee(func) : function(string) {
    return func.test(string);
  };
});

_.filter(['abc', 'def'], /ef/);
// => ['def']
10.4.11.5. _.property(path)

Creates a function that returns the value at path of a given object 

var objects = [
  { 'a': { 'b': 2 } },
  { 'a': { 'b': 1 } }
];

_.map(objects, _.property('a.b'));
// => [2, 1]

_.map(_.sortBy(objects, _.property(['a', 'b'])), 'a.b');
// => [1, 2]

10.4.12. lighthouse npm:lighthouse

https://github.com/GoogleChrome/lighthouse 

# install chromium browser
sudo apt install chromium-browser

npm i -g lighthouse

# chrome-flags
# --no-sandbox is required when user is root
# See chrome:cli 
lighthouse --chrome-flags="--headless --disable-gpu" https://github.com

# examples without --chrome-flags to quickly demo options

# output format and file name. html is default. json and csv.
lighthouse --output json
lighthouse --output html --output-path ./report.html

# extension is ignored. Result 2 files myfile.report.json and myfile.report.html
lighthouse --output json --output html --output-path ./myfile.json

lighthouse --output json --output html
# saves `./<HOST>_<DATE>.report.json` and `./<HOST>_<DATE>.report.html`

lighthouse --output-path=~/mydir/foo.out --save-assets
# saves `~/mydir/foo.report.html`
# saves `~/mydir/foo-0.trace.json` and `~/mydir/foo-0.devtoolslog.json`

10.4.13. font-awesome npm:font-awesome

  • npm install font-awesome --save
  • node_modules/font-awesome/fonts/fontawesome-webfont.woff
  • node_modules/font-awesome/fonts/fontawesome-webfont.woff2
  • node_modules/font-awesome/fonts/fontawesome-webfont.ttf

10.4.14. prismjs npm:prismjs

  • npm install prismjs --save
  • node_modules/prismjs/
    prism.js
    not minified. To minify run gulp. Default includes these languages:
    markup
    xml, html, mathml, svg
    (no term)
    css
    (no term)
    clike
    javascript
    js
    components/prism-scss.min.js
    more components
    (no term)
    themes/prism.css

10.4.15. bootstrap npm:bootstrap

  • node_modules/node_modules/bootstrap/dist/js/bootstrap.min.js
  • node_modules/jquery/dist/jquery.min.js
  • node_modules/popper.js/dist/umd/popper.min.js
git clone https://github.com/twbs/bootstrap.git
npm install

# compile and you will get css and js folders in /dist directory
npm run dist

# jQuery and popper.js
npm install --save jquery popper.js

10.4.16. browser-sync npm:browser-sync

Windows requires node-gyp It creates localhost:3000 which serves content from proxy server with <script async>...</script> inserted right after <body> HTML. Refer to this GitHub repo

Reload CSS without refreshing the page
.pipe( browserSync.stream() )

https 

function browsersync() {
    browserSync.init({

    // For more options
    // @link http://www.browsersync.io/docs/options/

    // Project URL.
    proxy: config.projectURL,

    // `true` Automatically open the browser with BrowserSync live server.
    // `false` Stop the browser from automatically opening.
    open: config.browserAutoOpen,



    https: {
        // both absolute and relative paths can be used
        key: '../../../cs-devops/lndo.site.key',
        cert: '../../../cs-devops/lndo.site.pem'
    },

    // Inject CSS changes.
    // Comment it to reload browser for every CSS change.
    injectChanges: config.injectChanges
    });

    // Use a specific port (instead of the one auto-detected by Browsersync).
    // port: 7000,
}

10.4.17. concurrently npm:concurrently

https://www.npmjs.com/package/concurrently
used in Angular
(no term)
npm install concurrently --save or npm install -g concurrently
concurrently "command1 arg" "command2 arg"
when installed globally
In package.json
"start": "concurrently \"node server.js\" \"gulp watch\""

10.4.18. concurrent-transform npm:concurrent-transform

npm i -D concurrent-transform Refer to npm:gulp-image-resize

10.4.19. gulp npm:gulp

https://github.com/gulpjs/gulp/blob/master/docs/README.md

If gulp was previously globally installed, remove it before installing gulp-cli 

npm rm --global gulp
npm install --global gulp-cli
gulp -v
# CLI version 2.0.1
# If gulp is installed locally, then
# Local version 4.0.0

Then install gulp locally npm install --save-dev gulp@latest

Gulp plugins
https://gulpjs.com/plugins/ https://github.com/alferov/awesome-gulp
gulp.task(
  'default',
  gulp.parallel(
    'styles',
    'vendorsJS',
    'customJS',
    'images',
    browsersync,
    function watchFiles() {
  gulp.watch( config.projectPHPWatchFiles, reload ); // Reload on PHP file changes.
  gulp.watch( config.styleWatchFiles, gulp.parallel( 'styles' ) ); // Reload on SCSS file changes.
  gulp.watch( config.vendorJSWatchFiles, gulp.series( 'vendorsJS', reload ) ); // Reload on vendorsJS file changes.
  gulp.watch( config.customJSWatchFiles, gulp.series( 'customJS', reload ) ); // Reload on customJS file changes.
  gulp.watch( config.imgSRC, gulp.series( 'images', reload ) ); // Reload on customJS file changes.
    }
  )
);
10.4.19.1. Example

gulp runs the default task. gulp <task> <othertask> 

npm install gulp-util --save-dev
npm install gulp-coffee --save-dev
npm install gulp-concat --save-dev
npm install gulp-browserify --save-dev
npm install gulp-compass --save-dev
npm install jquery --save-dev
npm install mustache --save-dev

gulpfile.js 

var gulp=require('gulp'),
    gutil = require('gulp-util'),
    coffee = require('gulp-coffee'),
    browserify = require('gulp-browserify'),
    compase = require('gulp-compass'),
    concat = require('gulp-concat');

// run ~gulp~ will run this task
gulp.task('default', function() {
  console.log('hello from gulp');
});

// gulp-util, run ~gulp log~
gulp.task('log', function() {
  gutil.log('Workflow starts');
});

var coffeeSources = ['components/coffee/tagline.coffee'];
var jsSources = [
  'components/scripts/1.js',
  'components/scripts/2.js'
];
var sassSources = ['components/sass/style.scss'];

gulp.task('coffee', function() {
  gulp.src(coffeeSources)
  .pipe(coffee({ bare: true })
     .on('error', gutil.log))
  .pipe(gulp.dest('components/scripts'));
});

gulp.task('js', ['coffee'], function() {
  gulp.src(jsSources)
  .pipe(concat('script.js'))
  .pipe(gulp.dest('builds/development/js'));
});

gulp.task('compass', function() {
  gulp.src(sassSources)
  .pipe(compoass({
        sass: 'components/sass',
        image: 'builds/development/images',
        style: 'expanded'
     })
     .on('error', gutil.log)
  )
  .pipe(gulp.dest('builds/development/css'));
});

gulp.task('watch', function() {
  gulp.watch(coffeeSources, ['coffee']);
});
10.4.19.2. Pass parameters from CLI to Gulp
//region Load CLI Parameters to Gulp
// e.g. `gulp task1 --a 123 --b "my string" --c`
// arg = { "a": "123", "b": "my string", "c": true }
const arg = (argList => {

    let arg = {}, a, opt, thisOpt, curOpt;
    for (a = 0; a < argList.length; a++) {

        thisOpt = argList[a].trim();
        opt = thisOpt.replace(/^\-+/, '');

        if (opt === thisOpt) {

            // argument value
            if (curOpt) arg[curOpt] = opt;
            curOpt = null;

        }
        else {

            // argument name
            curOpt = opt;
            arg[curOpt] = true;

        }

    }

    return arg;

})(process.argv);
//endregion
10.4.19.3. gulp.src, gulp.dest
  • gulp.src(globs[, options])
  • string or array. node-glob
  • options to pass to node-glob

    • negate 

      // negate can be used
gulp.src(['client/*.js', '!client/b*.js', 'client/bad.js'], function() {});

    • Multiple files 

        var files = [
      'node_modules/jquery/dist/jquery.min.js',
      'node_modules/bootstrap/dist/js/bootstrap.min.js',
      'node_modules/popper.js/dist/umd/popper.min.js'
  ];

  gulp.task('vendorjscopy', function() {
  return gulp.src(files)
    .pipe(gulp.dest('./public/js')
});

    • Only one base is possible for an array of files 

        /*
some/path/example/app/js/app.js
some/path/example/vendor/js/vendor.js
some/path/example/vendor/lib/js/lib.js
  */

  gulp.src('some/path/**/js/*.js')
  .pipe(gulp.dest('output'));

  // base is anything before the first ** which is some/path
  // result

  /*
output/example/app/js/app.js
output/example/vendor/js/vendor.js
output/example/vendor/lib/js/lib.js
  */

  // hard set base
  gulp.src('some/path/**/js/*.js', {base:'.'})
  .pipe(gulp.dest('output'));

  // result
  /*
output/some/path/example/app/js/app.js
output/some/path/example/vendor/js/vendor.js
output/some/path/example/vendor/lib/js/lib.js
  */

  // Does not work!
  gulp.src(
  ['source1/examples/**/*.html',
   'source2/examples/**/*.html'],
  { base: ['source1/', 'source2/']}   // Doesn't work. Needs to be a string.
  ).pipe(gulp.dest('dist'));


  // Instead
  var merge = require('merge-stream');
  gulp.task('default', function() {
  merge(gulp.src('source1/examples/**/*.html', {base: 'source1/'}),
    gulp.src('source2/examples/**/*.html', {base: 'source2/'}))
      .pipe(gulp.dest('dist'));
  });
    • current working directory
10.4.19.4. gulp-debug

See what files are run through Gulp pipeline

10.4.19.5. gulp-plumber

npm install --save-dev gulp-plumber

By design, Node stream will stop accepting incoming data, if error event was raised. You can see it in stream.js:103 - cleanup function will deattach ondata handler from source (which in our case is gulp.src) and coffee plugin will stop receiving files although, the rest of the files can be compiled. 

gulp.src('coffee/**/*.coffee')
  .pipe(gulpPrefixer('// Copyright 2014 (C) Aswesome company'))
  .on('error', gutil.log)
  .pipe(coffee())
  .on('error', gutil.log)
  .pipe(gulp.dest('js/'));

Use gulp-plumber 

var plumber = require('gulp-plumber');
var coffee = require('gulp-coffee');

gulp.src('./src/*.ext')
    .pipe(plumber())
    .pipe(coffee())
    .pipe(gulp.dest('./dist'));
10.4.19.6. gulp-remember
var gulp = require('gulp'),
    header = require('gulp-header'),
    footer = require('gulp-footer'),
    concat = require('gulp-concat'),
    cache = require('gulp-cached'),
    remember = require('gulp-remember');

var scriptsGlob = 'src/**/*.js';

gulp.task('scripts', function () {
  return gulp.src(scriptsGlob)
  .pipe(cache('scripts')) // only pass through changed files, 'scripts' is any cache name of your choice
  .pipe(header('(function () {')) // do special things to the changed files...
  .pipe(footer('})();')) // for example, add a stupid-simple module wrap to each file
  .pipe(remember('scripts')) // add back all files (initially in gulp.src) to the stream. 'scripts' is any cache name of your choice
  .pipe(concat('app.js')) // do things that require all files
  .pipe(gulp.dest('public/'))
});
10.4.19.7. gulp-cache

refer to gulp-remember

10.4.19.8. gulp-sort

Sort files in stream by path or any custom sort comparator

npm install gulp-sort --save-dev 

var sort = require('gulp-sort');

// default sort
gulp.src('./src/js/**/*.js')
    .pipe(sort())
    .pipe(gulp.dest('./build/js'));

// pass in a custom comparator function
gulp.src('./src/js/**/*.js')
    .pipe(sort(customComparator))
    .pipe(gulp.dest('./build/js'));

// sort descending
gulp.src('./src/js/**/*.js')
    .pipe(sort({
     asc: false
    }))
    .pipe(gulp.dest('./build/js'));

// sort with a custom comparator
gulp.src('./src/js/**/*.js')
    .pipe(sort({
    comparator: function(file1, file2) {
        if (file1.path.indexOf('build') > -1) {
        return 1;
        }
        if (file2.path.indexOf('build') > -1) {
        return -1;
        }
        return 0;
    }
    }))
    .pipe(gulp.dest('./build/js'));

// sort with a custom sort function
var stable = require('stable');
gulp.src('./src/js/**/*.js')
    .pipe(sort({
    customSortFn: function(files, comparator) {
        return stable(files, comparator);
    }
    }))
    .pipe(gulp.dest('./build/js'));
10.4.19.9. gulp-concat

Concat files using operating system's newLine 

var concat = require('gulp-concat');

gulp.task('scripts', function() {
  return gulp.src('./lib/*.js')
    .pipe(concat('all.js'))
    .pipe(gulp.dest('./dist/'));
});

// in order
gulp.task('scripts', function() {
  return gulp.src(['./lib/file3.js', './lib/file1.js', './lib/file2.js'])
    .pipe(concat('all.js'))
    .pipe(gulp.dest('./dist/'));
});
10.4.19.10. gulp-rename
var rename = require("gulp-rename");

// rename via string
gulp.src("./src/main/text/hello.txt")
  .pipe(rename("main/text/ciao/goodbye.md"))
  .pipe(gulp.dest("./dist")); // ./dist/main/text/ciao/goodbye.md

// rename via function
gulp.src("./src/**/hello.txt")
  .pipe(rename(function (path) {
    path.dirname += "/ciao";
    path.basename += "-goodbye";
    path.extname = ".md";
  }))
  .pipe(gulp.dest("./dist")); // ./dist/main/text/ciao/hello-goodbye.md

// rename via hash
gulp.src("./src/main/text/hello.txt", { base: process.cwd() })
  .pipe(rename({
    dirname: "main/text/ciao",
    basename: "aloha",
    prefix: "bonjour-",
    suffix: "-hola",
    extname: ".md"
  }))
  .pipe(gulp.dest("./dist")); // ./dist/main/text/ciao/bonjour-aloha-hola.md

Rename dest file name Any source *-gulp.css will have dest file *-gulp-build.css 

var gulp = require('gulp');

gulp.task('autoprefixer', function () {
  var postcss = require('gulp-postcss');
  var sourcemaps = require('gulp-sourcemaps');
  var rename = require('gulp-rename');
  var autoprefixer = require('autoprefixer');

  return gulp.src('**/*-gulp.css', {base: '.'})
    .pipe(sourcemaps.init())
    .pipe(postcss([autoprefixer()]))
    .pipe(sourcemaps.write('.'))
    .pipe(rename(function (path) {
  path.basename += '-build';
    }))
    .pipe(gulp.dest('.'));
});
10.4.19.11. gulp-sourcemaps
  • https://www.npmjs.com/package/gulp-sourcemaps
  • Another sourcemap file (endfile.css.map) is generated on top of the result file (endfile.css)
  • All plugins between sourcemaps.init() and sourcemaps.write() need to have support for gulp-sourcemaps
  • the path is relative to the destination
var gulp = require('gulp');
var plugin1 = require('gulp-plugin1');
var plugin2 = require('gulp-plugin2');
var sourcemaps = require('gulp-sourcemaps');

gulp.task('javascript', function() {
  gulp.src('src/**/*.js')
    .pipe(sourcemaps.init())
  .pipe(plugin1())
  .pipe(plugin2())
    .pipe(sourcemaps.write())
    .pipe(gulp.dest('dist'));
});
10.4.19.12. gulp-filter
const gulp = require('gulp');
const uglify = require('gulp-uglify');
const filter = require('gulp-filter');

gulp.task('default', () => {
    // Create filter instance inside task function
    const f = filter(['**', '!*src/vendor'], {restore: true});

    return gulp.src('src/**/*.js')
    // Filter a subset of the files
    .pipe(f)
    // Run them through a plugin
    .pipe(uglify())
    // Bring back the previously filtered out files (optional)
    .pipe(f.restore)
    .pipe(gulp.dest('dist'));
});
10.4.19.13. gulp-sass npm:gulp-sass

Doc Options are the same as node-sass 

var scssStream = gulp.src( config.styleSRC )
        .pipe( sourcemaps.init() )
        .pipe(
            sass({
                errLogToConsole: config.errLogToConsole,
                outputStyle: config.outputStyle,
                precision: config.precision
            })
        )
        .on( 'error', sass.logError )
        .pipe(concat('app.scss'));
  1. Add extra path for @import
    • includePaths is related to gulpfile.js or the file where sass is run. Default is empty array []
    • Say assets/scss/main.scss has @import subfolder/a.scss
      • If file exists assets/scss/subfolder/a.scss, use it. If not, go to next step
      • If file exists assets/scss/supporting/fallback/subfolder/a.scss, use it
    var scssStream = gulp.src( config.styleSRC )
    .pipe( sourcemaps.init() )
    .pipe(
    sass({
        errLogToConsole: config.errLogToConsole,
        outputStyle: config.outputStyle,
        precision: config.precision,
        includePaths: ['assets/scss/supporting/fallback/']
    })
    )
    .on( 'error', sass.logError )
    .pipe(concat('app.scss'));
10.4.19.14. gulp-merge-media-queries
var mmq = require('gulp-merge-media-queries');

gulp.task('mmq', function () {
  gulp.src('src/**/*.css')
    .pipe(mmq({
  log: true
    }))
    .pipe(gulp.dest('dist'));
});
10.4.19.15. gulp-line-ending-corrector

It converts CRLF \r\n to LF \n 

var lec = require( 'gulp-line-ending-corrector' );

// .pipe(lec())
// .pipe(lec({verbose:true, eolc: 'LF', encoding:'utf8'}))
10.4.19.16. gulp-uglifycss

Minify CSS using UglifyCSS

10.4.19.17. gulp-postcss npm:gulp-postcss Doc
npm i -D autoprefixer
npm i -D gulp-postcss

gulp.task('autoprefixer', function () {
  var postcss      = require('gulp-postcss');
  var sourcemaps   = require('gulp-sourcemaps');
  var autoprefixer = require('autoprefixer');

  return gulp.src('./src/*.css')
    .pipe(sourcemaps.init())
    .pipe(postcss([ autoprefixer() ]))
    .pipe(sourcemaps.write('.'))
    .pipe(gulp.dest('./dest'));
});

Load multiple PostCSS plugins, refer to npm:autoprefixer for options 

var postcss = require('gulp-postcss');
var gulp = require('gulp');
var autoprefixer = require('autoprefixer');
var cssnano = require('cssnano');

gulp.task('css', function () {
    var plugins = [
    autoprefixer({browsers: ['last 1 version']}),
    cssnano()
    ];
    return gulp.src('./src/*.css')
    .pipe(postcss(plugins))
    .pipe(gulp.dest('./dest'));
});
  1. npm:postcss:options
10.4.19.18. gulp-uglify

It uses UglifyJS2 to minify JavaScript files

10.4.19.19. gulp-babel

https://www.npmjs.com/package/gulp-babel https://babeljs.io/docs/en/options 

.pipe(
  babel({
    presets: [
  [
    'env', // Preset which compiles ES6 to ES5.
    {
      targets: { browsers: config.BROWSERS_LIST } // Target browser list to support.
    }
  ]
    ]
  })
)
10.4.19.20. gulp-responsive

https://github.com/mahnunchik/gulp-responsive

  1. Installation
    • Requires global node-gyp and dev dependency is sharp
    • For JPEG, PNG, WebP and TIFF
    npm install --save-dev gulp-responsive

# you may need to run this if there is still error, but I didn't need it
# npm -g install npm@latest
# https://github.com/lovell/sharp/issues/615
10.4.19.21. gulp-image-resize npm:gulp-image-resize
# make sure imagemagick is installed
apt list -a --installed imagemagick

# graphicsmagick is said to be faster than imagemagick
apt install graphicsmagick

npm install --save-dev gulp-image-resize

# Imagick config
#   - By default, it prevents the conversion from PDF to PNG. Comment out this line to relax this policy
<policy domain="coder" rights="none" pattern="PDF" />

# or
<policy domain="coder" rights="read|write" pattern="PDF" />

gulp.task('img-resize-max-width', function () {

    const imgs = gFilter('**/*.{jpg,JPG,jpeg,JPEG,png,PNG}', {restore: true});

    var maxw=1440;

    return gulp.src('src/**/*')
    .pipe(imgs)
    .pipe(gImageResize({
        width: maxw,
        height: 0,
        imageMagick:true, // default uses graphicmagick
        interlace: true
    }))
    .pipe(imgs.restore)
    .pipe(gulp.dest('dist'));
});

Use npm:concurrent-transform 

var parallel = require("concurrent-transform");
var os = require("os");

gulp.task("parallel", function () {
  gulp.src("src/**/*.{jpg,png}")
    .pipe(parallel(
  imageResize({ width : 100 }),
  os.cpus().length
    ))
    .pipe(gulp.dest("dist"));
});
10.4.19.22. gulp-imagemin
10.4.19.23. gulp-svg-sprite

https://github.com/jkphl/svg-sprite https://github.com/jkphl/gulp-svg-sprite

npm i -D gulp-svg-sprite

There're 5 modes :: css, view, defs, symbol, stack

10.4.19.24. gulp-notify

It sends notification on operating system level. 

.pipe( notify({ message: 'TASK: "styles" Completed!', onLast: true }) );

10.4.20. grunt

Install grunt globally and locally 

npm install grunt-cli -g
npm install grunt-cli --save-dev

npm install grunt-contrib-jshint --save-dev
npm install grunt-contrib-less --save-dev
npm install grunt-autoprefixer --save-dev
npm install grunt-browserify --save-dev
npm install grunt-contrib-watch --save-dev

npm install jquery

// GruntFile.js
module.exports = function(grunt) {

  grunt.initConfig({
    jshint: {
  files: ["*.js", "lib/*.js", "test/*.js"],
  options: {
    esnext: true,
    globals: {
      jQuery: true
    }
  }
    },
    less: {
  production: {
    files: {
      "public/css/style.css": ["less/*.less"]
    }
  }
    },
    autoprefixer: {
  single_file: {
    src: "public/css/style.css",
    dest: "public/css/style.css"
  }
    },
    browserify: {
  client: {
    src: ["app-client.js"],
    dest: "public/js/bundle.js"
  }
    },
    watch: {
  css: {
    files: ["less/*.less"],
    tasks: ["css"]
  },
  scripts: {
    files: ["app-client.js", "lib/*.js"],
    tasks: ["jshint", "browserify"]
  }
    }
  });

  grunt.loadNpmTasks("grunt-contrib-jshint");
  grunt.loadNpmTasks("grunt-contrib-less");
  grunt.loadNpmTasks("grunt-autoprefixer");
  grunt.loadNpmTasks("grunt-browserify");
  grunt.loadNpmTasks("grunt-contrib-watch");

  grunt.registerTask("css", ["less", "autoprefixer"]);
  grunt.registerTask("js", ["browserify"]);

  grunt.registerTask("default", ["jshint", "css", "js"]);
};

// ./app-client.js
var $ = require("jquery");
var printTerms = require("./lib/printTerms");

// jQuery code


// run grunt
grunt

// open another session and run
grunt watch

10.4.21. webpack

https://angular.io/guide/webpack ng:cli:webpack

10.4.21.1. Basics
mkdir webpack-demo
cd webpack-demo
npm init -y
npm install webpack webpack-cli --save-dev
10.4.21.2. Multiple bundles
entry: {
  app: 'src/app.ts',
  vendor: 'src/vendor.ts'
},

output: {
  filename: '[name].js'
}
10.4.21.3. Rules
module: {

 rules: [
   {
     test: /\.ts$/,
     loader: 'awesome-typescript-loader'
   },
   {
     test: /\.css$/,
     loaders: 'style-loader!css-loader'
   }
 ]

}
10.4.21.4. Plugins
const { UglifyJsPlugin } = require('webpack').optimize;

plugins: [
  new UglifyJsPlugin({
    // config options
  })
]

Webpack, the plugins, and the loaders are also installed as packages. They are listed in the updated packages.json.

  1. terser-webpack-plugin
    • https://github.com/webpack-contrib/terser-webpack-plugin
    • npm i terser-webpack-plugin -D

    • webpack.config.js 

      const TerserPlugin = require("terser-webpack-plugin");

module.exports = {
    optimization: {
        minimize: true,
        minimizer: [new TerserPlugin({
            terserOptions: {
                keep_fnames: true,
            },
        })],
    },
};
10.4.21.5. resolve

Most import statements don't mention the extension. Tell Webpack to resolve extension-less file requests by looking for matching files with .ts extension or .js extension. 

"resolve": {
    "extensions": [
      ".ts",
      ".js"
    ],
    "modules": [
      "./node_modules",
      "./node_modules"
    ],
    "symlinks": true
}
10.4.21.6. webpack.(common|dev|prod).js

webpack.common.js 

var webpack = require('webpack');
var HtmlWebpackPlugin = require('html-webpack-plugin');
var ExtractTextPlugin = require('extract-text-webpack-plugin');
var helpers = require('./helpers');

module.exports = {
  entry: {
    'polyfills': './src/polyfills.ts',
    'vendor': './src/vendor.ts',
    'app': './src/main.ts'
  },

  resolve: {
    extensions: ['.ts', '.js']
  },

  module: {
    rules: [
      {
        test: /\.ts$/,
        loaders: [
          {
            loader: 'awesome-typescript-loader',
            options: { configFileName: helpers.root('src', 'tsconfig.json') }
          } , 'angular2-template-loader'
        ]
      },
      {
        test: /\.html$/,
        loader: 'html-loader'
      },
      {
        test: /\.(png|jpe?g|gif|svg|woff|woff2|ttf|eot|ico)$/,
        loader: 'file-loader?name=assets/[name].[hash].[ext]'
      },
      {
        test: /\.css$/,
        exclude: helpers.root('src', 'app'),
        loader: ExtractTextPlugin.extract({ fallbackLoader: 'style-loader', loader: 'css-loader?sourceMap' })
      },
      {
        test: /\.css$/,
        include: helpers.root('src', 'app'),
        loader: 'raw-loader'
      }
    ]
  },

  plugins: [
    // Workaround for angular/angular#11580
    new webpack.ContextReplacementPlugin(
      // The (\\|\/) piece accounts for path separators in *nix and Windows
      /angular(\\|\/)core(\\|\/)@angular/,
      helpers.root('./src'), // location of your src
      {} // a map of your routes
    ),

    new webpack.optimize.CommonsChunkPlugin({
      name: ['app', 'vendor', 'polyfills']
    }),

    new HtmlWebpackPlugin({
      template: 'src/index.html'
    })
  ]
};

webpack.dev.js extends webpack.common.js 

  var webpackMerge = require('webpack-merge');
  var ExtractTextPlugin = require('extract-text-webpack-plugin');
  var commonConfig = require('./webpack.common.js');
  var helpers = require('./helpers');

  module.exports = webpackMerge(commonConfig, {
    devtool: 'cheap-module-eval-source-map',

    output: {
      path: helpers.root('dist'),
      publicPath: '/',
      filename: '[name].js',
      chunkFilename: '[id].chunk.js'
    },

    plugins: [
      new ExtractTextPlugin('[name].css')
    ],

    devServer: {
      historyApiFallback: true,
      stats: 'minimal'
    }
  });
#+END_EXAMPLE
The HtmlWebpackPlugin, added in webpack.common.js, uses the ~publicPath~ and ~filename~ in ~output~ to generate appropriate <script> and <link> tags into the index.html.

~output~ bundles in the dist folder, the dev server keeps all bundles in memory; it doesn't write them to disk. You won't find any files in the dist folder, at least not any generated from this development build.

Until the environment variable is set to production (webpack.DefinePlugin)
webpack.prod.js
#+BEGIN_EXAMPLE
  var webpack = require('webpack');
  var webpackMerge = require('webpack-merge');
  var ExtractTextPlugin = require('extract-text-webpack-plugin');
  var commonConfig = require('./webpack.common.js');
  var helpers = require('./helpers');

  const ENV = process.env.NODE_ENV = process.env.ENV = 'production';

  module.exports = webpackMerge(commonConfig, {
    devtool: 'source-map',

    output: {
      path: helpers.root('dist'),
      publicPath: '/',
      filename: '[name].[hash].js',
      chunkFilename: '[id].[hash].chunk.js'
    },

    plugins: [
      new webpack.NoEmitOnErrorsPlugin(),
      new webpack.optimize.UglifyJsPlugin({ // https://github.com/angular/angular/issues/10618
        mangle: {
          keep_fnames: true
        }
      }),
      new ExtractTextPlugin('[name].[hash].css'),
      new webpack.DefinePlugin({
        'process.env': {
          'ENV': JSON.stringify(ENV)
        }
      }),
      new webpack.LoaderOptionsPlugin({
        htmlLoader: {
          minimize: false // workaround for ng2
        }
      })
    ]
  });

Angular CLI set ENV in this way. The EnvironmentPlugin is shorthand for using the DefinePlugin on process.env keys. 

const { EnvironmentPlugin } = require('webpack');
"plugins": [
new EnvironmentPlugin({
    "NODE_ENV": "production"
  }),
]

webpack.prod.js addtional plugins

NoEmitOnErrorsPlugin
stops the build if there is an error.
UglifyJsPlugin
minifies the bundles.
ExtractTextPlugin
extracts embedded css as external files, adding cache-busting hash to the filename.
DefinePlugin
use to define environment variables that you can reference within the application.
LoaderOptionsPlugins
to override options of certain loaders.
10.4.21.7. Use of webpack.(common|dev|prod).js

src/index.html 

<!DOCTYPE html>
<html>
  <head>
    <base href="/">
    <title>Angular With Webpack</title>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
  </head>
  <body>
    <my-app>Loading...</my-app>
  </body>
</html>

src/main.ts 

import { platformBrowserDynamic } from '@angular/platform-browser-dynamic';
import { enableProdMode } from '@angular/core';

import { AppModule } from './app/app.module';

if (process.env.ENV === 'production') {
  enableProdMode();
}

platformBrowserDynamic().bootstrapModule(AppModule);

polyfills.ts is a good place to configure browser environment for production or development 

import 'core-js/es6';
import 'core-js/es7/reflect';
require('zone.js/dist/zone');

if (process.env.ENV === 'production') {
  // Production
} else {
  // Development and test
  Error['stackTraceLimit'] = Infinity;
  require('zone.js/dist/long-stack-trace-zone');
}

src/vendor.ts 

// Angular
import '@angular/platform-browser';
import '@angular/platform-browser-dynamic';
import '@angular/core';
import '@angular/common';
import '@angular/http';
import '@angular/router';

// RxJS
import 'rxjs';

// Other vendors for example jQuery, Lodash or Bootstrap
// You can import js, ts, css, sass, ...

Highlights

  • There are no <script> or <link> tags in the index.html. The HtmlWebpackPlugin inserts them dynamically at runtime
  • The AppComponent in app.component.ts imports the application-wide css with a simple import statement
  • The AppComponent itself has its own html template and css file. WebPack loads them with calls to require(). Webpack stashes those component-scoped files in the app.js bundle too. You don't see those calls in the source code; they're added behind the scenes by the angular2-template-loader plug-in
  • The vendor.ts consists of vendor dependency import statements that drive the vendor.js bundle. The application imports these modules too; they'd be duplicated in the app.js bundle if the CommonsChunkPlugin hadn't detected the overlap and removed them from app.js

10.4.22. PostCSS, postcss-cli cssnano autoprefixer

All PostCSS plugins, online catalog

Install npm package for a specific build tool e.g. npm:gulp-postcss And install individual PostCSS plugin which is separate npm package

npm:postcss:options Most PostCSS runners accept two parameters:

  • An array of plugins.
  • An object of options.

Common options:

  • syntax: an object providing a syntax parser and a stringifier.
  • parser: a special syntax parser (for example, SCSS).
  • stringifier: a special syntax output generator (for example, Midas).
  • map: source map options.
  • from: the input file name (most runners set it automatically).
  • to: the output file name (most runners set it automatically).

#+BEGIN_SRC js var gulp = require('gulp'); var postcss = require('gulp-postcss'); var nested = require('postcss-nested'); var sugarss = require('sugarss');

gulp.task('default', function () { var plugins = [nested]; return gulp.src('in.sss') .pipe(postcss(plugins, { parser: sugarss })) .pipe(gulp.dest('out')); }); #+END_SRC js

10.4.22.1. postcss-cli npm:postcss-cli

npm i -g|-D postcss-cli 

npm i postcss-cli autoprefixer
npx postcss *.css --use autoprefixer -d build/
10.4.22.2. autoprefixer npm:autoprefixer

Required by npm:gulp-postcss

It doesn't remove duplicate css rules and removes unwanted prefixes in source files.

  1. browserslist npm:postcss:browerslist

    package.json

    https://github.com/ai/browserslist#queries Default 

    {
...,
  "browserslist": [
    "> 0.5%",
    "last 2 versions",
    "Firefox ESR",
    "not dead"
  ],
...
}
  2. Comment to turn off prefixing

    Autoprefixer is disabled for a specific rule b 

    a {
    transition: 1s; /* it will be prefixed */
}

b {
    /* autoprefixer: off */
    transition: 1s; /* it will not be prefixed */
}

    /* autoprefixer: off */
@supports (transition: all) {
    /* autoprefixer: on */
    a {
        /* autoprefixer: off */
    }
}

    For Sass, add ! so that /*! autoprefixer: off */

  3. Options

    autoprefixer({ cascade: false })

    browsers (array)
    list of browsers query (like last 2 versions), which are supported in your project. We recommend to use browserslist config or browserslist key in package.json, rather than this option to share browsers with other tools. See Browserslist docs for available queries and default value.
    env (string)
    environment for Browserslist.
    cascade (boolean)
    should Autoprefixer use Visual Cascade, if CSS is uncompressed. Default: true
    add (boolean)
    should Autoprefixer add prefixes. Default is true.
    remove (boolean)
    should Autoprefixer [remove outdated] prefixes. Default is true.
    supports (boolean)
    should Autoprefixer add prefixes for @supports parameters. Default is true.
    flexbox (boolean|string)
    should Autoprefixer add prefixes for flexbox properties. With "no-2009" value Autoprefixer will add prefixes only for final and IE versions of specification. Default is true.
    grid (boolean)
    should Autoprefixer add IE prefixes for Grid Layout properties. Default is false.
    stats (object)
    custom usage statistics for > 10% in my stats browsers query.

    Plugin object has info() method for debugging purpose.

    You can use PostCSS processor to process several CSS files to increase performance.

  4. Debug

    Show selected browsers and css properties will be prefixed npx autoprefixer –info

10.4.22.3. cssnano npm:cssnano

Build on top of PostCSS, minify css

10.4.22.4. precss npm:precss

Use Sass-like markup and staged CSS features in .css file. npm install precss –save-dev

10.4.23. co-fs, co nodejs:co-fs nodejs:co

Default fs doesn't support Promise or Generator Function 

npm install co-fs
npm install co

var fs = require('co-fs');
co(function* () {
  var data = yield fs.readFile('./data1.json', 'utf-8');
  // fs.readFile is a promise by using co-fs
  console.log(data);
});

Before co@4.0.0, co() returned a thunk, which is a callback, after it returns a promise Chain 

co(function* () {
  var result = yield Promise.resolve(true);
  return result;
})
.then(function (value) {
  console.log(value);
}, function (err) {
  console.error(err.stack);
});

Convert co-generator function into a regular function which returns a promise 

var fn = co.wrap(function* (val) {
  return yield Promise.resolve(val);
});

fn(true).then(function (val) {
  // do something
});

Yieldables are

  • promises
  • thunks (functions)
  • array (parallel execution)
  • objects (parallel execution)
  • generators (delegation)
  • generator functions (delegation)

Resolve multiple promises in parallel 

co(function* () {
  var a = Promise.resolve(1);
  var b = Promise.resolve(2);
  var res = yield [a,b];
}).catch(onerror);

function onerror(err) {
  console.error(err.stack);
}

10.4.24. http, https nodejs:http

It's sync request.

10.4.24.1. request
var https = require('https');
var fs = require('fs');

var options = {
  hostname: "en.wikipedia.org",
  port: 443,
  path: "/wiki/George_Washington",
  method: "GET"
};

var req = https.request(options, function(res) {
  var responseBody = "";
  console.log(`Server Status: ${res.statusCode}`);
  console.log("Response Headers: %j", res.headers);
  res.setEncoding("UTF-8");
  res.once("data", function(chunk) {
    console.log(chunk);
  });

  res.on("data", function(chunk) {
    console.log(`--chunk-- ${chunk.length}`);
    responseBody += chunk;
  });

  res.on("end", function() {
    fs.writeFile("george-w.html", responseBody, function(err) {
      if (err) throw err;
    });
    console.log("File downloaded");
  });

});

// Set timeout
req.on('socket', function(socket) {
  socket.setTimeout(8000);
  socket.on('timeout', () => {
    req.abort();
    console.log('timeout');
  });
});

req.on("error", function(err) {
  if (err.code === 'ECONNRESET') {
    console.log("Timeout occurs");
  }
  console.log(`problem: ${err.message}`);
});

req.end();
10.4.24.2. server

Simple HTTP server which gives responses 

var http = require('http');
var fs = require('fs');
var path = require('path');
var jsonFile = require('./data/inventory'); //inventory.json
// jsonFile is a json object, not string

var server = http
  .createServer(function (req, res) {
    if (req.url === '/') {
      fs.readFile("./public/index.html", "UTF-8", function (err, html) {
        res.writeHead(200, {"Content-Type": "text/html"});
        // res.write(html); res.end();
        res.end(html);
      });
    }
    else if (req.url.match(/.css$/)) {
      var cssPath = path.join(__dirname, 'public', req.url);
      var fileStream = fs.createReadStream(cssPath, "UTF-8");
      res.writeHead(200, {"Content-Type": "text/css"});
      fileStream.pipe(res);
    }
    else if (req.url.match(/.jpg$/)) {
      var imgPath = path.join(__dirname, 'public', req.url);
      var imgStream = fs.createReadStream(imgPath); // binary
      res.writeHead(200, {"Content-Type": "image/jpeg"});
      imgStream.pipe(res);
    }
    else if (req.url === 'getallorders') {
      res.writeHead(200, {"Content-Type": "text/json"});
      res.end(JSON.stringify(jsonFile));
    }
    else if (req.url === 'instock') {
      listInStock(res);
    }
    else if (req.url === 'form') {

      if (res.method === "GET") {
        res.writeHead(200, {"Content-Type": "text/html"});
        fs.createReadStream("./public/form.html", "UTF-8").pipe(res);
      }
      else if (req.method === "POST") {
        var body = "";
        // body will be a url encoded string with key-value pairs
        req.on("data", function(chunk) {
          body += chunk;
        });

        req.on("end", function() {
          res.writeHead(200, {"Content-Type": "text/html"});
          res.end(`${body}`);
        });
      }

    }
    else {
      res.writeHead(200, {"Content-Type": "text/plain"});
      res.end("404 File Not Found");
    }
  })
  .listen(3000);

function listInStock(res) {
  var inStock = jsonFile.filter(function(item) {
    return item.avail === "In Stock";
  });
  res.end(JSON.stringify(inStock));
}

10.4.25. cors

You don't need to config Nginx! 

npm install cors

var express = require('express');
var cors = require('cors');
var app = express();

// Enable CORS for all routes/requests
app.use(cors());

app.get('/products/:id', function (req, res, next) {
  res.json({msg: 'This is CORS-enabled for all origins!'});
});

// Enable for one route
// Don't put cors() in app.route()
app.get('/products/:id', cors(corOptions), function (req, res, next) {
  res.json({msg: 'This is CORS-enabled for all origins!'})
})

// Options
var corsOptions = {
  origin: 'http://example.com',
  optionsSuccessStatus: 200
}

// Multiple domains
var whitelist = ['http://example1.com', 'http://example2.com']
var corsOptions = {
  origin: function (origin, callback) {
    if (whitelist.indexOf(origin) !== -1) {
      callback(null, true)
    } else {
      callback(new Error('Not allowed by CORS'))
    }
  }
}

// use the same cors(corsOptions)

10.4.26. xml2js

https://github.com/Leonidas-from-XIV/node-xml2js 

npm install --save xml2js

var parseString = require('xml2js').parseString;
var xml = "<root>Hello xml2js!</root>"
parseString(xml, function (err, result) {
    console.dir(result);
});

10.4.27. casual nodejs:casual

10.4.28. crypto - Built-in nodejs:crypto

10.4.28.1. HMAC

SSL certificate changes from SHA1 to SHA256 which is SHA2 algorithm.

One ASCII character is 8 bits. One UTF-8 character is between 8 bits to 32 bits.

One ASCII character can be represented as 2 hex characters (UTF-8, etc.)

Output size of SHA256 is 256 bits, which is 256/8*2 = 64 hex characters Output size of SHA1 is 160, which is 40 hex characters

The secret/key of SHA256 should at least contain 256 bits = 256/8 bytes = 32 bytes. But I use 128 bits.. Which is 128/8*2 = 32 hex characters. 

var crypto = require('crypto')
 , text = 'hello'
 , key = 'abcdeg' // better to have 32 hex characters, but length could be any
 , hash

let hmac = crypto.createHmac('sha256', key);
hmac.write(text);
hmac.end();
let hash = hmac.read().toString('hex');
10.4.28.2. encrypt, decrypt

Refer to php:openssl_decrypt

All AES, regardless 128/256 or key length, the output size depends on the input size (floor(Input size / 16 bytes) + 1) * 16 bytes 

const crypto = require('crypto');

const KEYAES = '64 hex characters'
const ALGORITHM = 'aes-256-cbc';

// store this keyAES256.toString('hex')
// convert hex to bytes
// keyAES256 = new buffer(keyAES256, 'hex');

function encrypt(plain_text) {
  var iv = crypto.randomBytes(16); // 16 text characters, 16 bytes or 32 hex characters
  // iv should be different every time

  var keyAES = new Buffer(KEYAES, 'hex');
  // var keyAES = crypto.randomBytes(32); // 32 text characters, 32 bytes or 64 hex characters
  // global KEYAES is keyAES.toString('hex')

  var cipher = crypto.createCipheriv(ALGORITHM, keyAES, iv);
  cipher.write('text to encrypt');
  cipher.end();

  var cipher_text = cipher.read().toString('hex');

  return cipher_text + '$' + iv.toString('hex');
}

function decrypt(cipher_text) {
  var cipher_blob = cipher_text.split("$");
  var ct = cipher_blob[0];
  var iv = new Buffer(cipher_blob[1], 'hex'); // get iv
  var keyAES = new Buffer(KEYAES, 'hex');

  decryptor = crypto.createDecipheriv(ALGORITHM, keyAES, iv);
  decryptor.write(ct, 'hex');
  decryptor.end();
  console.log(decryptor.read().toString('utf-8'));
}
10.4.28.3. generate random characters for unique id

let id = require('crypto').randomBytes(10).toString('hex')

10.4.29. body-parser

parse POST data. Refer to nodejs:express npm install body-parser

var express = require('express');

10.4.30. httpster

npm install httpster -g
httpster -p 3000 -d ./public/

# Or httpster is installed locally
./node_modules/httpster/bin/httpster -p 3000 -d ./public/

10.4.31. express nodejs:express

10.4.31.1. Sample
// my-express.js
var express = require('express');
var app = express();
var bodyParser =  require('body-parser');

app.set('port', process.env.PORT || 3000);

var skierTerms = [{},{}]; // javascript array/object. global to the project/app. Only reset when app is restarted.
app.set('appData', skierTerms);

// app.use() adds middlewares which run first

app.use(bodyParser.json()); // any data sent to this app (API) as .json, the data will be parsed
app.use(bodyParser.urlencoded({ extended: true })); // also parse url encoded data
// false uses the `querystring` library which means the data could be string or array
// true uses the `qs` library which means the data could be any type

// log every request
app.use(function(req, res, next) {
  console.log(`${req.method} request for '${req.url}' - ${JSON.stringify(req.body)}`);
  next(); // eventually, res.send('output');, so here is next
});

// default is index.html, server static files
app.use(express.static(__dirname));

app.use('/message', function(req, res) {
  console.log('user requested endpoint');
  res.send('response is hello');
});

app.use(require('./routes/index'));
app.use(require('./routes/dict-api'));

app.listen(app.get('port'), function() {
  // optional callback
});

// visit localhost:3000/message
// node my-express.js
// PORT=4000 node my-express.js

// ./routes/dict-api.js
var express = require('express');
var router = express.Router();

router.use(bodyParser.json()); // any data sent to this app (API) as .json, the data will be parsed
router.use(bodyParser.urlencoded({ extended: false })); // also parse url encoded data

// same domain
router.get('/dict-api', function(req, res) {
  var skierTerms = req.app.get('appData');
  res.json(skierTerms); // send json variable, no need to stringify
});

router.post('/dict-api', function(req, res) {
  var skierTerms = req.app.get('appData');
  skierTerms.push(req.body);
  req.app.set('appData', skierTerms);
  res.json(skierTerms); // send json variable, no need to stringify
});

router.delete('dict-api/:term', function(req, res) {
  var skierTerms = req.app.get('appData');
  skierTerms = skierTerms.filter(function(def) {
    return def.term.toLowerCase() !== req.params.term.toLowerCase();
  });
  req.app.set('appData', skierTerms);
  res.json(skierTerms);
});

module.exports = router;
10.4.31.2. Routing
  1. Callbacks

    Instead of next() which skips to the next callback, next('route') can bypass all the remaining route callbacks of the current app.METHOD() or router.METHOD(). next('router') to skip the rest of the router's middleware functions the router instance 

    app.get('/example/a', function (req, res) {
  res.send('Hello from A!')
})

app.get('/example/b', function (req, res, next) {
  console.log('the response will be sent by the next function ...')
  next()
}, function (req, res) {
  res.send('Hello from B!')
})

var cb0 = function (req, res, next) {
  console.log('CB0')
  next()
}

var cb1 = function (req, res, next) {
  console.log('CB1')
  next()
}

var cb2 = function (req, res) {
  res.send('Hello from C!')
}

app.get('/example/c', [cb0, cb1, cb2])

var cb0 = function (req, res, next) {
  console.log('CB0')
  next()
}

var cb1 = function (req, res, next) {
  console.log('CB1')
  next()
}

app.get('/example/d', [cb0, cb1], function (req, res, next) {
  console.log('the response will be sent by the next function ...')
  next()
}, function (req, res) {
  res.send('Hello from D!')
})
  2. Response methods
    1. res.send res.sendFile
      // Content-Type is automatically sent when res.send is used
res.send(new Buffer('whoop'));
res.send({ some: 'json' }); // Array or Object, Express responds with JSON
res.send('<p>some html</p>'); // Content-Type text/html
res.status(404).send('Sorry, we cannot find that!');
res.status(500).send({ error: 'something blew up' });

// Send an octet stream :: http://expressjs.com/en/4x/api.html#res.sendFile
// res.sendFile(path [, options] [, fn])

res.sendFile(__dirname + '/views/index.html' );
    2. res.write res.end

      Quickly ends the response without any data. 

      // send multiple responses using res.write
res.write('1');
res.write('2');
res.setHeader('Content-Type', 'text/html');
res.end();
// res.status(404).end();
  3. Parameters
    // Route path: /users/:userId/books/:bookId
// Request URL: http://localhost:3000/users/34/books/8989
// req.params: { "userId": "34", "bookId": "8989" }

app.get('/users/:userId/books/:bookId', function (req, res) {
  res.send(req.params)
})

// Route path: /user/:userId(\d+)
// Request URL: http://localhost:3000/user/42
// req.params: {"userId": "42"}

// Request URL: /mv?id=abc
// req.query: {id: "abc"}

    The name of route parameters must be made up of “word characters” ([A-Za-z0-9_]).

    - and . are literal characters - and .

    For using * as regex, use {0,} for Express 4.x, use * for Express 5.

10.4.31.3. Error-handling middleware

Error-handling middleware are defined last. A built-in error handler is added at the end. If next(err) is called and there're no custom error handler, it'll be handled by the built-in error handler. In non-production environment (NODE_ENV=production), stack trace is printed.

For a custom error handler, if the headers have already been sent to the client, you'll need to call next(err) to let the built-in error handler to close the connection and fail the request.

next(err) can be only called once otherwise there is an error 

function errorHandler(err, req, res, next) {
  if (res.headersSent) {
    return next(err)
  }
  res.status(500)
  res.render('error', {error: err})
}

Simple sample: 

// 4 arguments
app.use(function (err, req, res, next) {
  console.error(err.stack);
  res.status(500).send('Error!');
})

Multiple error handlers 

// /index.js
app.use(logErrors)
app.use(clientErrorHandler)
app.use(errorHandler)

function logErrors (err, req, res, next) {
  console.error(err.stack)
  next(err)
}

function clientErrorHandler (err, req, res, next) {
  if (req.xhr) {
    res.status(500).send({ error: 'Something failed!' })
  } else {
    next(err)
  }
}

function errorHandler (err, req, res, next) {
  res.status(500)
  res.render('error', { error: err })
}
10.4.31.4. Built-in middleware
  1. express.static for serving static assets
    // syntax: express.static(root, [options])
// serve static files in /public directory
app.use(express.staic('public'));
10.4.31.5. View Engine (ejs)
npm install ejs --save

// app.js
app.set('view engine', 'ejs');
app.set('view', './views'); // default

app.locals.siteTitle = 'Website Name'; // available to all views
// persist through the life of the app

// ./routes/index.js
router.get('/', function(req,res) {
  var data = req.app.get('appData');
  var pagePhotos = [];

  data.speakers.forEach(function() {
    pagePhotos = pagePhotos.concat(item.artwork);
  });

  res.render('index', {
    pageTitle: 'Home',
    artwork: pagePhotos,
    pageID: 'home'
  });
});

// ./views/index.ejs
<!-- HTML Code -->
<%= siteTitle %>
<%= pageTitle %>

<body id="<%= pageID %>">

// if the variable is HTML, output as HTML
<%- varHTML %>

<% if (typeof pageTitle !== "undefined") { %>
  <script src="abc.js"></script>
<% } %>

<% if (artwork.length > 0) { %>
  <% for (i=0; i< artwork.length; i++) { %>
  <img src="/images/artwork/<%= artwork[i] %>" alt="Artwork <%= i %>">
  <% } %>
<% } %>

<% include partials/template/header.ejs %>
// partial can use the parent view variables

10.4.32. sails

npm install sails -g
mkdir sails
cd sails
// Create a sails project with backend only
sails new --no-front-end

// create an api
sails generate api user

// 2 files are created
// ./sails/api/controllers/UserController.js
// ./sails/api/models/User.js

Modify models/User.js, what attributes a user can have and the datatype? 

module.exports = {
  attributes: {
    name: 'string'
  }
}

Start sails, default port is 1337 :: sails lift

Perform CRUD by visiting url 

localhost:1337/user/create?name=John
localhost:1336/user

10.4.33. koa

ES6 

npm install koa

// koa-endpoint-demo.js
var koa = require('koa');
var app = new koa();
app.use(function* () {
  this.body = 'hello in the body tag';
});

app.listen(3000);

Run it in harmony 

node --harmony koa-endpoint-demo.js

10.4.34. Promise

10.4.34.1. Basics

Promise is native, refer to nodejs:bluebird 

var promise = new Promise(function(res, rej) {
  resolve();
  // or
  // reject();
});

promise.then(function() {
  console.log('then');
})
.catch(function() {
  console.error('failed');
});
10.4.34.2. http server example
const parseString = require('xml2js').parseString;

route.get('/auth/:site/:emailkey', function(req, res) {
  try {
    getUserInfo(req.params.site, req.params.emailkey, res);
  }
  catch (e) {
    res.json(e.message);
  } 
});

function getUserInfo(site, emailkey, res) {
  httpsRequest(httpsOptions)
    .then(function(body) {
      parseString(body, (err, result) => {
        res.json(result);
      });
    });
}

function httpsRequest(params) {
  return new Promise(function(resolve, reject) {
    let req = https.request(params, function(res) {
      let body = '';

      res.on('data', (chunk) => {
        body += chunk;
      });

      res.on('end', () => {
        promisesParser(body)
          .then((result) => {
            resolve(body);
          })
          .catch((err) => {
            reject(err);
          });
      });

    }); // https.request

    req.end();

  }); // Promise
}

function promiseParser(string) {
   return new Promise(function(resolve, reject) {
     parseString(string, function(err, result) {
       if (err) {
         reject(err);
       }
       else {
         resolve(result);
       }
     });
   });
}

10.4.35. bluebird nodejs:bluebird

// npm install bluebird

var fs = require('fs');
var Promise = require('bluebird');
Promise.promisifyAll(fs);
// append Async after every fs object (e.g. methods)
fs.readFileAsync('./data1.json')
  .then(JSON.parse)
  .then(function (val) {
    console.log(val);
  })
  .catch(SyntaxError, function(e) {
    console.error("invalid json in file");
  })
  .catch(function(e) {
    console.error("unable to read file");
  });

10.4.36. lite-server npm:lite-server

A web server that refreshes when file is updated
https://www.npmjs.com/package/lite-server
// npm i -g lite-server

10.4.37. node-dev, nodemon, reload

node-dev restart automatically when there's a change 

npm install node-dev -g
npm install nodemon -g
node-dev app.js
# vs node app.js

# nodemon can watch on a list of file types or ignore a file/folder
nodemon -e js,jade app --watch /another/folder/to/watch --ignore feedback.json --ignore aFolder/

In Vagrant, use nodemon -L index.js for --legacy-watch restarting server

Create a WebSocket on the client and reload the browser when Express server is reloaded 

npm install -g reload
npm install reload --save

var reload = require('reload');

// app.js
// after app.listen();
reload(server, app);

// In each route, add reload client js file.
router.get('/', function(req, rs) {
  res.write('<script src="/reload/reload.js"></script>');
});

10.4.38. jshint

Check javascript syntax 

npm install jshint -g
jshint app.js

// In app.js, insert a comment at top to give extra instructions
// Use ES6 standard
/* jshint esnext: true */

10.4.39. mongoose nodejs:mongoose

  • Refer to jsSandbox:lynda-graphql_essential_training

10.4.40. sequelize nodejs:sequelize

  • ORM for MySQL, MariaDB, SQLite, Postgres and MSSQL
  • Refer to jsSandbox:lynda-graphql_essential_training

10.4.41. ws

Refer to js:ws for client 

npm install ws --save

var WebSocketServer = require('ws');
var wss = new WebSocketServer({port:3000});
wss.on('connection', function(ws) {
  wss.on('message', function(msg) {
    if (message === 'exit') {
      ws.close();
    }
    else {
      // broadcast
      wss.clients.forEach(function(client) {
        client.send(msg);
      });
    }
  });
  ws.send('Welcome to cyber chat');
});

10.4.42. socket.io

npm install socket.io --save

var express = require('express');
var http = require('http'); // socket.io requires http
var app = express();
var server = http.createServer(app).listen(3000);
var io = require('socket.io')(server); // change http server to a socket server
// Or io can be attached later: require('socket.io')();

app.use(express.static('./public'));

// socket is attached later
// io.attach(server);

io.on('connection', function(socket) {
  socket.on('chat', function(msg) {
    socket.broadcast.emit('message', message);
  });
  socket.emit('message', 'Welcome to cyber chat');
});

// socket.io-client.min.js
// main.js
var socket = io("http://localhost:3000");
socket.on('disconnect', function() {
  // do something
});
socket.on('connect', function() {
  // do something
});
socket.on('message', function(msg) {
  console.log(msg);
});

socket.emit('chat', input.value);

10.4.43. mocha, chai, nock, rewire, sinon

npm install mocha -g
npm install chai --save-dev

// nock is an http mocking and expectations library
npm install nock --save-dev

// rewire adds special setter and getter to modules to modify their values
// https://www.npmjs.com/package/rewire
npm install rewire --save-dev
npm install sinon --save-dev

// ./test/tools-spec.js
var expect = require('chai').expect;
var tools = require('../lib/tools'); // tools.js
var nock = require('nock');

// Test function printName()
describe('printName()', function() {
  // Test 1
  it("should print the last name first", function() {
    var results = tools.printName({ first: "Li", last: "Banks" });
    expect(results).to.equal("Banks, Alex");
  });
});

// Test async function loadWiki()
describe('loadWiki()', function() {
  this.timeout(5000); // default timeout is 2 seconds
  // Test 1
  it("Load Abraham Lincon's wiki page", function(done) {
    tools.loadWiki({ first: 'Abraham', last: 'Lincooln'}, function(html) {
      expect(html).to.be.ok;
      done(); // if done() is not run, test will fail
    });
  });
});
var expect = require('chai').expect;
var tools = require('../lib/tools'); // tools.js
var nock = require('nock');
var rewire = require('rewire');

var order = rewire('../lib/order');
// instead of require(), use rewire() which can change data later

// Test function printName()
describe('printName()', function() {
  // Test 1
  it("should print the last name first", function() {
    var results = tools.printName({ first: "Li", last: "Banks" });
    expect(results).to.equal("Banks, Alex");
  });
});

// Test async function loadWiki()
describe('loadWiki()', function() {
  //this.timeout(5000); // default timeout is 2 seconds

  before(function() {
    // before runs before all tests in this block
    // mock server, save testing time
    nock('https://en.wikipedia.org')
      .get('/wiki/Abraham_Lincoln')
      .reply(200, `--html-code--`);
  });

  // Test 1
  it("Load Abraham Lincon's wiki page", function(done) {
    tools.loadWiki({ first: 'Abraham', last: 'Lincooln'}, function(html) {
      //expect(html).to.be.ok;
      expect(html).to.equal('--html-code--');
      done(); // if done() is not run, test will fail
    });
  });
});

describe('Ordering Items', function() {
  beforeEach(function() {
    // beforeEach runs before each test in this block
    this.testData = [
      {sku: "AAA", qty: 10},
      {sku: "BBB", qty: 10}
    ];
    // modify data for testing purpose
    order.__set__("inventoryData", this.testData);
  });

  it('order an item when there are enough in stock', function(done) {
    order.orderItem('BBB', 3, function() {
      // orderItem subtracts 3 from inventory
      done();
    });
  });

});
var expect = require('chai').expect;
var tools = require('../lib/tools'); // tools.js
var nock = require('nock');
var rewire = require('rewire');
var sinon = require('sinon');

var order = rewire('../lib/order');
// instead of require(), use rewire() which can change data later

// Test function printName()
describe('printName()', function() {
  // Test 1
  it("should print the last name first", function() {
    var results = tools.printName({ first: "Li", last: "Banks" });
    expect(results).to.equal("Banks, Alex");
  });
});

// Test async function loadWiki()
describe('loadWiki()', function() {
  //this.timeout(5000); // default timeout is 2 seconds

  before(function() {
    // before runs before all tests in this block
    // mock server, save testing time
    nock('https://en.wikipedia.org')
      .get('/wiki/Abraham_Lincoln')
      .reply(200, `--html-code--`);
  });

  // Test 1
  it("Load Abraham Lincon's wiki page", function(done) {
    tools.loadWiki({ first: 'Abraham', last: 'Lincooln'}, function(html) {
      //expect(html).to.be.ok;
      expect(html).to.equal('--html-code--');
      done(); // if done() is not run, test will fail
    });
  });
});

describe('Ordering Items', function() {
  beforeEach(function() {
    // beforeEach runs before each test in this block
    this.testData = [
      {sku: "AAA", qty: 10},
      {sku: "BBB", qty: 10}
    ];
    // modify data for testing purpose
    order.__set__("inventoryData", this.testData);

    this.console = {
      // overwrite a function with a fake function
      log: sinon.spy()
    };

    order.__set__('console', this.console);

    this.warehouse = {
      packageAndShip: sinon.stub().yields(1098765)
    };

    order.__set__('warehouse', this.warehouse);

  });

  it('order an item when there are enough in stock', function(done) {
    var _this = this;

    order.orderItem('BBB', 3, function(done) {
      // orderItem subtracts 3 from inventory

      expect(_this.console.log.callCount).to.equal(2);

      done();
    });
  });

  describe('Warehouse interaction', function() {
    beforeEach(function() {
      this.callback = sinon.spy();
      order.orderItem("BBB", 2, callback);
    });

    it('receives a tracking number', function() {
      expect(this.callback.calledWith(1098765)).to.equal(true);
    });

    it('calls packageAndShip with the correct sku and quantity', function() {
      expect(this.warehouse.packageAndShip.calledWith("BBB",2)).to.equal(true);
    });

  });

});

// ./lib/tools.js
var https = require('https');
module.exports = {
  printName(person) {
    return `${person.last}, ${person.first}`;
  },
  loadWiki(person, callback) {
    var url = `https://en.wikipedia.org/wiki/${person.first}_${person.last}`;
    https.get(url, function(res) {
      var body = "";
      res.setEncoding('UTF-8');
      res.on("data", function(chunk) {
        body += chunk;
      });
      res.on("end", function() {
        callback(body);
      });
    });
  }
};

// ./lib/order.js
var inventoryData = require('../data/inventory');
var warehouse = require('./warehouse');

function findItem(sku) {
  var i = inventoryData.map(item => item.sku).indexOf(sku);
  if (i === -1) {
    console.log(`Item - ${sku} not found`);
    return null;
  } else {
    return inventoryData[i];
  }
}

function isInStock(sku, qty) {
  var item = findItem(sku);
  return item && item.qty >= qty;
}

function order(sku, quantity, complete) {
  complete = complete || function () {};
  if (isInStock(sku, quantity)) {
    console.log(`ordering ${quantity} of item # ${sku}`);
    warehouse.packageAndShip(sku, quantity, function (tracking) {
      console.log(`order shipped, tracking - ${tracking}`);
      complete(tracking);
    });
    return true;
  } else {
    console.log(`there are not ${quantity} of item '${sku}' in stock`);
    return false;
  }
}

module.exports.orderItem = order;

// run mocha
mocha

mocha command not found, add this to package.json 

"scripts": {
  "test": "mocha -R spec"
}

-R spec is used to console.log inside tests it()

10.4.44. supertest, cheerio

npm install supertest --save-dev
npm install cheerio --save-dev

// ./app.js is an express app

// ./test/app-spec.js
var request = require("supertest");
var expect = require('chai').expect;
var cheerio = require("cheerio");
var rewire = require('rewire');
var app = rewire('../app');

describe("Dictionary App", function () {

    it("Loads the home page", function(done) {
        request(app).get("/").expect(200).end(function(err, res) {
          var $ = cheerio.load(res.text); // jQuery DOM!
          var pageHeading = $("body>h1:first-child").text();
          expect(pageHeading).to.equal("Skier Dictionary");
          done();
        });
    });

    describe("Dictionary API", function () {

        beforeEach(function () {

                this.defs = [
                {
                    term: "One",
                    defined: "Term One Defined"
                },
                {
                    term: "Two",
                    defined: "Term Two Defined"
                }
            ];

            app.__set__("skierTerms", this.defs);
        });

        it("GETS dictionary-api", function(done) {
            var defs = this.defs;
            request(app).get("/dictionary-api").expect(200).end(function(err, res) {
                var terms = JSON.parse(res.text);
                expect(terms).to.deep.equal(defs);
                done();
            });
        });

        it("POSTS dictionary-api", function(done) {
            request(app)
               .post("/dictionary-api")
               .send({ "term": "Three", "defined": "Term Three Defined"})
               .expect(200)
               .end(done);
        });

        it("DELETES dictionary-api", function(done) {
            request(app)
               .delete("/dictionary-api/One")
               .expect(200)
               .end(done);
        });

    });

});

10.4.45. istanbul

Code coverage report (testing). 

npm install istanbul -g

# at ./
istanbul cover _mocha

# ./coverage/lconv-report/index.html

10.4.46. svgo nodejs:svgo

https://github.com/svg/svgo 

// I found windows has permission issue if it's installed globally
npm install svgo

// optimize the svg
./node_modules/svgo abc.svg

// output the svg to data uri URL encoded
// enc, unenc, base64
svgo abc.svg --datauri enc

// increase precision number of decimal points
svgo a.svg --precision=8

// output result STDOUT. if not, the file will be overwritten
svgo abc.svg --datauri enc -o -

nodejs:svgo:move styles

Usually it's good in IE11 but if it still doesn't work, refer to svg:ie11

AI already makes .svg and you want to move styles to inline svgo abc.svg –enable=inlineStyles –config '{ "plugins": [ { "inlineStyles": { "onlyMatchedOnce": false } }] }'

Move style attribute from each node to attributes :: style="width:…;height:…;" to width="…" height="…" svgo abc.svg –enable=convertStyleToAttrs

–enable=PLUGIN : Enable plugin by name, "–enable={PLUGIN3,PLUGIN4}" for multiple plugins

Batch convert all *.svg files in a folder and output to another folder svgo -f ../path/to/folder/with/svg/files -o ../path/to/folder/with/svg/output

10.4.47. serve

default port
3000
To change port in command line
serve -s build -p 3001
(no term)
npm i -g serve

10.5. Custom Module

// my-module.js
// Export one property at a time
exports.myText = 'hello';

// Variables local to the module will be private

// my-square.js, square module
// Root of this module is a function
// Or you want to export a complete object in one assignment instead of building it one property at a time
// Assign it to `module.exports` instead of `exports`
module.exports = (width) => {
  return {
   area: () => width ** 2
  }
}

// module-demo.js
var myModule= require('./my-module.js');
const square = require('./my-square.js');
const mySquare = square(2);
console.log(myModule.myText);
console.log(`The area of my square is ${mySquare.area()}`);

// node module-demo.js

# Initiate the custom module, adding package.json file
npm init

# --save flag will add/remove dependency to package.js
npm install cors --save
npm remove cors --save

# delete ./node_modules and run this to install all dependencies
npm install

Call local functions 

var _foo = function () {
  return ('foo');
}
var _bar = function () {
  return _foo();
}
module.exports = {
  foo: _foo,
  bar: _bar
};

10.5.1. package.json, node-inspector

  • npm install node-inspector -g
  • refer to devDependencies
  • Run start in scripts
  • Run a script in scripts
  • Property
    main
    a file to include when another user to include the module by require('ski-dictionary')
    devDependencies
    npm install --production or when NODE_ENV environment variable is production, npm will not install modules listed in devDependencies
{
  "name": "ski-dictionary",
  "version": "1.0.0",
  "description": "A collection of skier terms and definitions",
  "main": "app.js",
  "scripts": {
    "predebug": "grunt",
    "debug": "open http://localhost:3000 & open http://localhost:8080/debug?port=5858",
    "prestart": "grunt",
    "start": "node app",
    "predev": "grunt",
    "dev": "open http://localhost:3000 & node-dev app & grunt watch"
  },
  "keywords": [
    "ski",
    "terms",
    "dictionary"
  ],
  "author": "Alex Banks",
  "license": "MIT",
  "dependencies": {
    "body-parser": "^1.14.1",
    "cors": "^2.7.1",
    "express": "^4.13.3",
    "jquery": "^2.1.4"
  },
  "devDependencies": {
    "grunt": "^0.4.5",
    "grunt-autoprefixer": "^3.0.3",
    "grunt-browserify": "^4.0.1",
    "grunt-contrib-jshint": "^0.11.3",
    "grunt-contrib-less": "^1.0.1",
    "grunt-contrib-watch": "^0.6.1"
  }
}

10.6. node-gyp node-gyp

Some plugins need node-gyp installed Open PowerShell with admin right 

npm install --global --production windows-build-tools
npm install --global node-gyp

windows-build-tools downloads and installs VC++ Build Tools and Python 2.7, configuring maching and npm appropriately.

The install is conflict-free meaning that they do not mess with existing installations of VS, C++ Build Tools (C++ runtime libraries) or Python.

10.7. NPM CLI

  • npm run-script <command> [--silent] [-- <args>...]

10.7.1. run-script nodejs:npm:cli:run-script

  • npm run-script <command> [--silent] <-- <args>...>]
  • alias
  • without command will return the list of scripts
  • npm run <a-script-in-scripts>
  • ">pass custom arguments to the command specified but not the command's pre or post script
  • https://docs.npmjs.com/misc/scripts Some of them:
    preinstall, [install, postinstall]
    npm install
    "install": "node-gyp rebuild"
    default if binding.gyp exists in root
    pretest, test, posttest
    npm test
    prestop, stop, poststop
    npm stop
    prestart, start, poststart
    npm start
    "start": "node server.js"
    default if server.js exists in root
    prerestart, restart, postrestart
    npm restart
  • npm run <stage>
  • Run a script from dependencies
10.7.1.1. postinstall
{
  "name": "...-...-...",
  "author": "...",
  "version": "2.0.2",
  "license": "GPL-3.0",
  "repository": {
    "type": "git",
    "url": "https://github.com/..."
  },
  "dependencies": {
    ...
  },
  "devDependencies": {
    ...
  },
  "scripts": {
    "postinstall": "npm run start",
    "start": "gulp",
    "styles": "gulp styles",
    "stylesRTL": "gulp stylesRTL",
    "vendorsJS": "gulp vendorsJS",
    "customJS": "gulp customJS",
    "images": "gulp images",
    "clearCache": "gulp clearCache",
    "translate": "gulp translate"
  }
}

10.7.2. install npm:cli:install

npm i abc
wihtout giving options, current version of npm will auto add it to dependency in package.json
(no term)
npm i -D eq. npm install --save-dev
npm i gist:mygistid
create a Gist on GitHub with package.json

10.8. Increase memory and NODE_OPTIONS nodejs:memory

  • Default memory is 512mb. Increase to 1gb
  • https://nodejs.org/api/cli.html#cli_node_options_options
  • For command line e.g. npm install
    • NODE_OPTIONS=--max-old-space-size=1024 npm install something
  • For node project
    • node --max_old_space_size=1024 main.js
  • For gulp
    • gulp yourTask --max_old_space_size=1024 or
    • node --max-old-space-size=1024 ./node_modules/.bin/gulp yourTask
  • Using a package and build a script

    • npm install --save-dev increase-memory-limit 

        // ...
"scripts": {
  "fix-memory-limit": "cross-env LIMIT=1024 increase-memory-limit"
},
"devDependencies": {
  "increase-memory-limit": "^1.0.3",
  "cross-env": "^5.0.5"
}
  // ...

10.9. yarn

10.9.1. Basics

npm install -g yarn

mkdir test
cd test
yarn init

# add a devDependency
yarn add --dev ava

# a yaml file `yarn.lock` is created

# install a global package
yarn global add rollup

10.9.2. ava 

mkdir test
touch test/index.test.js

package.json, add this 

"scripts": {
    "test": "ava"
}

index.test.js 

import test from 'ava';

test('reverse', t => {
    t.is('hello', 'olleh', 'The strings to not match')
    t.is('goodbye','dadfa')
})

10.10. TS: gulp-util is deprecated

  • Older version of a pkg might use gulp-util pkg which is deprecated
  • npm ls gulp-util
  • Say gulp-sass is using it, update it
    See installed gulp-sass version
    npm ls gulp-sass
    See latest version avaible
    npm view gulp-sass versions --json
    Update it and update package.json
    npm i -D gulp-sass or npm i --save gulp-sass
    Reinstall
    rm -rf node_modules && rm -f package-lock.json && npm install

10.11. TS: pathspec

Command failed: C:\Program Files\Git\mingw64\bin\git.EXE checkout 4.0
error: pathspec '4.0'

Change package.json from "gulp": "gulpjs/gulp#4.0" to "gulp": "4.0"

10.12. TS: Unmet dependencies

  • rm -rf node_modules rm package-lock.json npm install
  • npm cache verify or npm cache clean in older NPM version
  • The problem might be
    • Fail to download the package, timed-out.
    • Manully install top-level modules npm i eslink-config-wordpress
    • or adjust the order of packages in package.json

10.13. TS: ENOSPC: System limit for number of file watchers reached

Increase the inotify max_user_watches limit For Debian echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

11. Prerender.io

https://github.com/prerender/prerender

Install middleware on .htaccess to rewrite search engine traffic to prerender server

example.com is your site and its .htaccess is 

# Change YOUR_TOKEN to your prerender token and uncomment that line if you want to cache urls and view crawl stats
# Change http://example.com (at the end of the last RewriteRule) to your website url

<IfModule mod_headers.c>
    #RequestHeader set X-Prerender-Token "YOUR_TOKEN"
</IfModule>

<IfModule mod_rewrite.c>
    RewriteEngine On

    <IfModule mod_proxy_http.c>
    RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator [NC,OR]
    RewriteCond %{QUERY_STRING} _escaped_fragment_

# Only proxy the request to Prerender if it's a request for HTML
    RewriteRule ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent|\.ttf|\.woff))(.*) http://service.prerender.io/http://example.com/$2 [P,L]
    </IfModule>
</IfModule>

Install prerender on prerender server 

git clone https://github.com/prerender/prerender.git
cd prerender
npm install
node server.js

Prerender will now be running on http://localhost:3000. If you wanted to start a web app that ran on say, http://localhost:8000, you can now visit the URL http://localhost:3000/http://localhost:8000 to see how your app would render in Prerender.

:3000/http://:8000 may return something that is 504. Don't worry it will work after Rewrite.

Change options in ./server.js

For better result, add this to every page and search engine will add ?_escaped_fragment_ to the URL 

<meta name="fragment" content="!">
html5 push state
URL http://example.com/user/1 becomes http://example.com/user/1?_escaped_fragment_=
hashbang
http://example.com/#!/user/1 becomes http://example.com/?_escaped_fragment_=/user/1

12. Social Media

12.1. WhatsApp

https://wa.me/?text= https://wa.me/15551234567/?text=

Old
whatsapp://send?text=
$temp_link = get_the_title().' : '.get_permalink().'?utm_source=whatsapp&utm_medium=social&utm_campaign=WhatsappShare';
$temp_link = 'https://wa.me/?text='.urlencode($temp_link);

12.2. Twitter

12.2.1. Compose link with prepopulated Tweet

12.2.3. Recent Tweets

https://publish.twitter.com/

Widget shows png image converted to jpg with 85% quality and black background color. Maybe add a transparent pixel and save the png to truecolor to avoid the conversion.

12.2.4. Twitter card twitter:card

12.2.5. TweetDeck twitter:tweetdeck

  • https://tweetdeck.twitter.com
  • Account @abc is created by abc@gmail.com, abc@gmail.com is the owner, one account has only one owner
    • May use a.bc@gmail.com to create another Twitter account, emails will also send to abc@gmail.com
    • can link other Twitter accounts as admin or contributor
      • Manage password, phone number and login verification settings
    • can link other accounts as admin or contributor, but Owner has to authorize
      • Tweet, Retweet, DM, like, etc., schedule Tweets, create lists and build collections
    • permissions like Admin but can't link other Twitter accounts
  • Everyone invited to TweetDeck Team has to use TweetDeck to post/use Twitter

12.2.6. Twitter Ads

  • Objective
    • Awarenes: promote Tweets and maximize reach
    • Engagement: promote Tweets and get more Retweets, likes and replies
    • Followers: promote account
    • Website clicks and App installs
  • Audience
    • Geographic
    • Followers of a notable account
    • Human interests
  • Bidding
    • Auto bidding
    • Pay for interaction: new follower, a click on your website
  • Budget: no minimum, daily budget
  • Creative
    • Call-to-actions
  • Twitter Ads Manager: ads.twitter.com

    • User management

      • Access Levels
      • Account administrator (AA)
        • Can create AA and AM
      • Ad Manager (AM)
        • Can't create users
      • Creative manager
      • Campaign analyst
      • Organic analyst: analytics.twitter.com
      • Partner audience manager

12.3. Facebook

12.3.1. Facebook for developers

12.3.2. Image Guidelines

12.3.3. Page

  • Settings
    • Page Roles
      Page Owner
      admins of a FB account can manage roles and other permissions on this Page
      Agencies
      assign approved permissions on this Page as a whole to all people inside an agency
      Existing Page Roles
      individuals

12.4. Instagram

  • Personal Profile
    • Private account
    • Linking to multiple FB Pages
  • Business Profile account
    • Create a personal/normal Insta account, login to Facebook which has permission to manage an FB Page
      • On Insta, Switch to Business Profile and select the FB Page
    • Access to Instagram Analytics
    • Promote posts as ads
    • Promote posts e.g. boost a post on FB Page
    • Contact Button near the top
    • Shopping + Checkout
    • require your approval to allow other Insta accounts to tag you as a branded content partner
    • Add links to Instagram Stories
    • Schedule and auto publish posts to Insta
    • DM quick replies
    • Can't go private on Instagram
  • Creator Profile
    • For influencers
    • Direct Messages filtering
    • Access creator-specific analytics including follow/unfollow metric and engagement stats
    • Shopping + Checkout
  • Photos and Videos Requirements

12.5. YouTube

Link to subscribe
https://www.youtube.com/channel/YOURHASEDCHANNELID?sub_confirmation=1
Recommended upload encoding settings
https://support.google.com/youtube/answer/1722171?hl=en&ref_topic=2888648 youtube:encoding

12.6. LinkedIn

12.6.1. LinkedIn Page

  • Add an admin
    • An existing page admin has to have first degree connection with the target person who is going to be added as admin

12.6.2. LinkedIn Marketing Solutions

  • User Role
    Account Manager
    manage users
    (no term)
    Campaign Manager
    (no term)
    Creative Manager
    (no term)
    Viewer
    (no term)
    Billing Admin
    • Each account needs at least one billing admin
    • Account creator is assigned as the billing admin
  • LinkedIn advertising hierarchy
    • account > campaign group > campaign > ad
      account
      currency, LinkedIn Company Page
      campaign group
      budget, start/end dates, status:active
      campaign
      define possible ad formats: Sponsored Content, Sponsored InMail and Text Ads
  • LinkedIn Insight Tag
    • A LinkedIn account can have multiple accounts and each account associates with one company page. Each Insight tag is unique to one account and thus one company page
    • An account can add multiple users. Each user under an account has a unique Insight tag
    • A tag can be on multiple websites and domains have to be verified on LinkedIn Campaign Manager
    • A tag can provide conversion tracking, learn audience and collect audience for remarketing

12.7. Buffer.com

  • Pricing

    Account Social Accounts Team Members Scheduled posts per social
    Free 3 0 10
    Business: Medium 50 10 2000
           
    Free
    3 social accounts, 0 team member, scheduled posts per social
  • For each social account, define post time

13. SEO & Marketing

13.3. Google My Business

  • Alternative
    • Bing places for business

    • Local directory management services

      • Yellow Pages
      • Superpages.com
        • Yelp
        • BBB.org
        • Manta.com
        • Yahoo yext.com
        • AngiesList.com
        • advicelocal.com
        • synup.com
        • brightlocal.com
        • Data Aggregators
      • InfoGroup.com
      • Acxiom.com
  • Insights
    • Direct Search and Discovery Search
    • Search views and Maps views
    • Visit your website, Request directions, Call you, View photos
  • Local Business Categories
  • https://developers.google.com/my-business/

13.4. Google Analytics google:ga

13.4.1. Limits, Versions, Accounts

  • GA sets and reads cookies for each unique domain e.g. www.example.com and dogtoys.example.com have 2 different cookies. Javascript document.domain
  • GA for FireBase is free and limited to track mobile apps
  • https://developers.google.com/analytics/devguides/collection/analyticsjs/limits-quotas
  • GA 360 is a paid per year service that includes a Service Level Agreement (SLA) that supports higher hit volumes
  • Sampling
    • Standard report always has unsampled data and if you add a segment, create a report, filter the report, then sampled data will be used
    • sampling occurs in Property. Default sample is 250K visits (sessions) and can be changed to 500K per property
    • sampling occurs in View. Default sample is 250k visits (sessions) and can be changed to 100M sessions per view
      • View Filter won't affect sampling
  • 360 can
    • use Google BigQuery
    • custom tables, unsampled reports
    • Roll-up reporting aggregates data from multiple Analytics properties and lets you see that data together in the same reports. e.g. website and mobile app stats
  • google:ga:limit:dimensions metrics
    GA
    20 and 20. Value of custom dimension has max length 150 bytes around 150 characters
    360
    200 and 200. 10 times more custom dimensions & metrics
  • Integration
    GA
    AdWords and AdSense. Import stats from AdWords to GA and GA exports remarketing audiences to AdWords
    360
    GA + DoubleClick Campaign Manager, DoubleClick Bid Manager, DoubleClick for Publishers, Salesforce Marketing Cloud
  • Custom Data Sources
    GA
    import
    360
    query-time import custom data sources (compare historic data and view the new imported data)
  • Standard Attribution Modeling Tool
    GA
    give some formula-based models for attributing conversions to channels (like first-touch, last-touch, time decay, etc.)
    360
    But which of those is the “right” model? GA360 includes a data driven attribution model, which actually uses an algorithm on your data to understand where different channels make the most impact
  • Funnel Reporting Options
    360
    Enhanced custom funnel reporting options. e.g. How many users view the page then scroll to 50% then scroll to the end of article then scroll to the end of page
  • An Analytics account (analytics account ID) can have up to 50 properties and each property can have up to 25 views
13.4.1.1. Hit limits
10 million hits per month per property
applies to Web Property / Property / Tracking ID (e.g. UA-xxx)
if go over
you will be contacted asking you to upgrade to 360 or implement client sampling to reduce the amount of data being sent to Google Analytics
GA 360
up to several billions
200,000 hits per user per day and 500 hits per session (universal analytics)
gtag.js, analytics.js, Android SDK, iOS SDK and Measurement Protocol
500 hits per session
for legacy tracking library (ga.js)
Session timeout
default 30 minutes. Min 1 minute and Max 4 hours
(no term)
Tracker objects
Start with 20 hits that are replenished at a rate of 2 hits per second
gtag.js and analytics.js tracker object. Applies to all hits except for ecommerce (item or transaction)
Start with 10 hits that are replenished at a rate of 1 hit per second
ga.js

Cut down hits per month

  • Separate traffic by setting up new property
  • Cut down spam traffic

13.4.2. Views, Filters, Segments

Admin > Account > Property > Views
Up to 25 views per property. 35 days to recover a deleted View
(no term)
Views collect data when they are created
(no term)
Views may have View Filters and changes on Views and Filters will affect future data only
(no term)
Under Views, there're a lot of reports (Standard and Custom). Apply Segments to reports to filter out data without destroying Views data
(no term)
Always have a View without Filters to capture all data
3 views
Unfiltered, Main and Test. Test new filters on Test View first and then apply to actual views
13.4.2.1. View Filter

A Filter is to filter out the types of traffic you don't want and save the data to your View.

  • Filter might take 24 hours to activate
  • Filter will limit the stats data from the moment it is fully activated
  • Filters are account-level. If filters are changed on View level, they are also changed across all properties of that account

In order to show only traffic that are on a set of pages, you can:

  • Figure out a Regex statement and use it as the filter that is inside the View > Default Segment ○ You can't save this filter. You have to do it manually every time you want the report ○ You can see the filtered results right away and you can see previous stats
  • Use JavaScript to submit Custom Variable on those pages and set to Page-level. ○ Create a new View with Admin > Property > View > Filter ○ You have to wait 24 hours to let the stats to be recorded from then ○ You can't see the previous stats

You can refer to this for constructing your ga:regex in Admin > Property > View > Filter or the normal filter.

Basic GA Filters

Custom Filter - Advanced

  • Extract 2 values from 2 fields (Field A and B), and output that to another field
  • The output field can be one of these
    • Custom Field
    • System field (e.g. Request URI, override the field value)
    • User Defined value
  • Request URI includes URL parameters but without domain and protocol
  • Custom Field 1 and 2 are used only in filters. Appears on GA report Audience > Custom > Custom Variables
  • User Defined value can be set by JavaScript and it's a single value. Appears on GA report Audience > Custom > User Defined
  • Example, only include traffic that URI has utm_source=MV_* and utm_medium=email

Advanced Filter

Field A
(utm_source=MV_[^&]*)
Field B
(utm_medium=email)
Output to Custom Field 1
$A1&$B1
(no term)
Setting Field 1 and 2 are required, Override Output Field

Custom Filter

  • Include, Custom Field 1, utm_source=[^&]*&utm_medium=email
13.4.2.3. Include domain names (Custom Filter)
Filter Type
Custom filter > Advanced
Field A
Hostname Extract A: (.*)
Field B
Request URI Extract: (.*)
Output To
Request URI Constructor: $A1$B1

13.4.3. Dimensions vs Metrics ga:dimensions ga:metrics ga:scope

  • Each cell shows the value in the corresponding metric column that also belong to the corresponding dimension row
    • e.g. metric is Goal completion and dimension is country. Table shows number of goal completions in each country
  • Which dimension value has the highest metric? e.g. Which page category (custom dimension) has the highest pageviews (built-in metrics)
  • Filters are only applied to dimensions
  • Dimensions are fields that can be grouped by in SQL
  • https://support.google.com/analytics/answer/2709828#scope
    Product
    value is applied to the product for which it has been set (Enhanced Ecommerce only)
    Hit
    Categorize actions e.g. pageview, event. A hit can associate with multiple products
    Session
    Categorize sessions, each session may have multiple hits. Timeout limit by default is 30 minutes. Max can go to 4 hours. Admin > Property
    User
    Categorize users. A cookie that has sessions
  • the true total of each metric might be larger than the top of the total shown in the top row. Percentage of shown in each cell is calculated against the true total of each metric/column. Each column in the top row total, I think, is the unique metric
    • e.g. Primary:Browser (user) and Secondary:Post Terms (hit)
  • Last hit's dimension send defines the hit, session and level dimension
  • Metric can only have Hit or Product level
    • Metrics can be set as KPI's to indicate macro or micro conversions
    • sum, ratio
    • Metrics are usually submitted using Event Hits and may associate with custom dimension
    • An example of a custom metric may be an Event Value of a certain Event
  • All standard Dimensions & Metrics Explorer in what scope
13.4.3.1. Custom dimensions and metrics ga:custom dimensions
  • https://support.google.com/analytics/answer/2709828
  • Custom Dimension can be used as secondary dimension in standard reports but as primary in Custom Reports and can be used as ga:segment
  • Allow you to combine Analytics data with non-Analytics data, e.g. CRM data. For example:
    • If you store the gender of signed-in users in a CRM system, you could combine this information with your Analytics data to see Pageviews by gender
    • If you're a game developer, metrics like "level completions" or "high score" may be more relevant to you than pre-defined metrics like Screenviews. By tracking this data with custom metrics, you can track progress against your most important metrics in flexible and easy-to-read custom reports
  • Custom dimensions can appear as primary dimensions in Custom Reports. You can also use them as Segments and secondary dimensions in standard reports.
  • Up to 20 custom dimensions and 20 custom metrics
13.4.3.2. Calculated Metric

Admin > Property > View > Calculated Metrics

13.4.4. Bounce rate

  • A bounce is counted when there's only one pageview in one user session
  • A bounce is calculated specifically as a session that triggers only a single request to the Analytics server, such as when a user opens a single page on your site and then exits without triggering any other requests to the Analytics server during that session
  • Bounce rate is single-page sessions divided by all sessions, or the percentage of all sessions on your site in which users viewed only a single page and triggered only a single request to the Analytics server
  • These single-page sessions have a session duration of 0 seconds since there are no subsequent hits after the first one that would let Analytics calculate the length of the session

13.4.5. Audience

13.4.5.1. User, New User, Session ga:identifiers
  • Refer to ga:gtag:config
  • UA-12315-12 (tid) where 12315 is Analytics account ID or Analytics account number
  • UA-12315-12 is the same as the tracking id
  • The total number of users for the requested time period. Long time cookie. Same browser and same device
  • The number of users whose session was marked as a first-time session
  • Short time cookie. Each session from a unique user will get its own incremental index starting from 1 for the first session. Subsequent sessions do not change previous session indices. For example, if a user has 4 sessions to the website, sessionCount for that user will have 4 distinct values of ‘1’ through ‘4’.
Cookie Expiration Description
_ga 2 yrs Client ID
_gid 24 hours Used to distinguish users
_gat 1 minute  
AMP_TOKEN 30 secs to 1 yr Client ID from AMP Client ID service.
_gac_<property-id> 90 days GA is linked with AdWords account, AdWords website conversion tags
     
  • _ga or _gid cookie example
    Value
    GA1.2.2033959936.1513880474
    Client ID
    2033959936.1513880474
    Domain
    e.g. .myweb.ca
13.4.5.2. Active user

Users that initiate session(s) in the last n days. Also called site reach, stickiness. 7-Day Active Users: the number of unique users who initiated sessions on your site or app from January 22 through January 28 (the last 7 days of your date range).

13.4.5.3. Cohort Analysis

Cohort analysis helps you understand the behavior of component groups of users apart from your user population as a whole. Examples of how you can use cohort analysis include to see how the behavior and performance of individual.

e.g. Group an audience based on their acquisition date and then compare behavior metrics over a series of weeks

Cohort type :: the only option now is Acquisition Date :: the first time a user is recognized as interacting with your content. When selected in the Cohort Analysis report, the cohorts are grouped based on when users started their first sessions.

Say metric is Pageviews and Date Range is 7 days. Today's April 12 For users visited the website for the first time on April 12-7 = April 5, they also visit n PageViews on Day 1 the next day April 6.

Metric User Retention :: The number of users in the cohort who returned in the Nth time period (day, week, month) divided by the total number of users in the cohort.

e.g. 3.02% of the users who visited the website for the first time on April 5, also visit the website on the next day, April 6 (Day 1).

13.4.5.4. Audiences
  • This report shows only audiences you create in Analytics. The new audience will have up to 30 days of data and you can use it in 24-48 hours
  • At any one time, you can have a maximum of 20 audiences published to Analytics. 2000 audiences per property
  • Audiences are available only in the view in which you create them
  • Audience can be published to only 1 destination of each type e.g. Google Ads or Display & Video 360, Optimize and Analytics. Once an audience is published to Google Ads/Google Marketing Platform, the destination cannot be changed
    • Audience includes Age, Gender or nay of the Interest dimensions, audience can only be published to Google Ads (Display) and Analytics
    • Audience includes Sequences, it cannot be published to Analytics
  • Search and Display audiences are backfilled with up to 30 days of data
  • Audiences that are based on Session Date dimension need 5 days before the session date
  • Audiences that are based on custom dimensions that use query-time import mode are not supported. Audience data is evaluated at processing time
  1. Create Audience ga:audience:create

    https://support.google.com/analytics/answer/6155470

  2. Preconfig new audience
    • Smart List: Let Google manage the audience for you
    • All users to your site or app who already have the necessary advertising cookies or mobile-advertising IDs
    • Any users who have conducted only one session on your site or app
    • Any users who have conducted more than one session on your site or app
    • Users who visited a specific section of my site/app
      • Click the edit icon, and enter the URL of a page or directory on your site, or a screen in your app. This option uses the contains match type, and matches any URL that contains the string you enter here.
      • If there are more than 1000 page/screen URLs for your site/app, then Analytics displays matches as you enter text only if matches are found within the first 1000 URLs. If there are no matches in the first 1000 URLs, then Analytics displays nothing. In this case, you can copy and paste the URL from a browser, or from some other source of URLs like a spreadsheet
    • Click the edit icon, and select a goal from the menu. This option requires that you have previously configured Analytics Goals
    • This is already configured to include any user with more than zero transactions
  3. Use Audiences
    • https://support.google.com/analytics/answer/7280979
    • After you create an audience and publish it to Analytics, it is available as a primary dimension in the Audience > Audiences report, as a secondary dimension in other reports, and as a dimension in segments, custom reports, and custom funnels.
13.4.5.5. Demographics ga:audience:demographics
Enable Demographics and Interests Report
https://support.google.com/analytics/answer/2819948
Enable Advertising Feautures (Advertising Reporting Features)
Admin > Property > Tracking Info > Data Collection
13.4.5.6. Interests ga:audience:interests
Enable Demographics and Interests Report
https://support.google.com/analytics/answer/2819948
Affinity
with passion or interests in an area but not necessarily looking to buy
In-Market Segments
looking to buy
(no term)
Other
13.4.5.7. User Explorer

Instead of aggregating users or user sessions, this shows each user/session's

  • Sessions
  • Total Session Duration / Sessions. The last session that a user stops navigating to another page/hit has 0 second duration. High bounce rate may cause this metric to be low
  • Bounce rate
  • Revenue
  • Transactions
  • Goal Conversion Rate
13.4.5.8. Benchmarking

You can choose from over 1600 industry categories, using a menu in the Benchmarking reports. You can further refine the data by geographic location and select from seven traffic size classifications, allowing you to compare your property against properties with similar traffic levels in your industry. For example, you can compare your property with all properties in the “All Hotels and Accommodations” industry in the United Kingdom that receive 500 to 1000 average daily sessions.

Compare in

  • Channels
  • Location
  • Devices
13.4.5.9. Users Flow

Segment does not work.. To show google/cpc, change the first dimension item from Country to Source/Medium = google/cpc

13.4.6. Acquisition ga:channel

  • Channel is the traffic source
    Change the Default Channel Grouping
    Admin > Property > View > Channel Settings > Channel Grouping > Default Channel Grouping
    • Direct
    • Search (Organic + Paid)
    • Referral
    • Email
    • Social
    • Organic Search
    • Paid Search
    • Other Advertising
    • Display
    Add a custom channel grouping
    Admin > Property > View > Custom Channel Grouping
    • e.g. define a channel to include traffic from a campaign containing the term January (e.g., January1, 2ndJanuary, January). Instead of specifying the rule as January (which will only return January), enter it as .*January.*
  • is more like channels, and source provides more specific about the medium
    • Every referral to a website also has a medium. Possible medium include:
      organic
      unpaid search
      cpc
      cost per click, i.e. paid search
      (no term)
      referral
      email
      the name of a custom medium you have created)
      none
      direct traffic has a medium of “none”
  • Every referral to a web site has an origin, or source. Possible sources include:
    google
    the name of a search engine
    facebook.com
    the name of a referring site
    spring_newsletter
    the name of one of your newsletters
    direct
    users that typed your URL directly into their browser, or who had bookmarked your site
  • When SSL search is employed, Keyword will have the value (not provided)
  • the name of the referring AdWords campaign or a custom campaign that you have created
  • a specific link or content item in a custom campaign. For example, if you have two call-to-action links within the same email message, you can use different Content values to differentiate them so that you can tell which version is most effective. AdWords Ad or extension name
  • search keywords
  • is case sensitive and spaces are converted to _
  • Refer to google:campaign-url-builder
  Email campaign Paid search campaign
Campaign Source newsletter1 yahoo
Campaign Medium email cpc
Campaign Term   the search term associated with this traffic
Campaign Content call_to_action_2  
Campaign Name productxyz productxyz
     
  • Use source/medium as google/cpc to have AdWords Search and Display Networks
  • To distinguish Paid Search and Display, use Default Channel Grouping
  • To distinguish Google Search and Search Partners, use Ad Distribution Network
  • Once AdWords account(s) is linked to GA, you can see relevant reports in Acquisition > AdWords

ga:dimensions

  • Accounts, Campaigns, Treemaps, Bid Adjustment, Keywords, Search Queries, Hour of Day, Destination URLs, Display Targeting, Video Campaigns, Shopping Campaigns

ga:metrics

  • Acquisition
    • Clicks, Cost, CPC, Users
  • Behavior
    • Bounce Rate, Pages/Session
  • Conversion by each goal
    • Conversion rate, Transactions, Revenue

For some dimension, you can filter by Device type on the top.

13.4.7. Behavior

  • How users navigate between pages
  • Speed, Search, Events
13.4.7.1. Average Time on Page ga:avg. time on page
  • The average length of time users spend viewing a page/specific set of pages
  • Because the last pageview of a user does not have a next pageview, so Google cannot calculate the last page duration. That's why the formula minus the Exit pageviews
  • It is usually higher than ga:avg. session duration because Exit pageviews is not counted in this metric. If it's lower, it might mean each session has a lot of pageviews
  • Time on Page / (Pageviews - Exits)
13.4.7.2. Content Grouping
Standard Report
Admin > Property > View > Content Grouping
(no term)
Content Grouping is not suited for grouping WordPress post categories as a post can have multiple categories and the same Content Grouping can have only one value
Methods to create a Content Group
in order. First matching method defines the content group
  • Modify the tracking code on each page
  • Extract pages with regex (Page URL, Page Title)
  • Create rules to include pages in a group (Page URL, Page Title, Screen name)
(no term)
Usually Content Grouping is submitted with pageview hit type

  • Behavior > Site Content > all reports

    • Content Group has to be set before or at the moment pageview hit is sent
    <script async src="https://www.googletagmanager.com/gtag/js?id=GA_TRACKING_ID"></script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());
  // original :: config and send a pageview
  //gtag('config', 'GA_TRACKING_ID');
  // change to
  gtag('config', 'GA_TRACKING_ID', {'content_group1': 'shoes'});

  // or
  // gtag('set', {'content_group1': ''});
  // gtag('config', 'GA_TRACKING_ID');
</script>

    <script>
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','https://www.google-analytics.com/analytics.js','ga');

  ga('create', 'UA-XXXXX-Y', 'auto');
  // set before pageview is sent
  ga('set', 'contentGroup1', 'my_group_name');
  ga('send', 'pageview');
</script>
  • It can be sent in other event types e.g. hit as well
    • Haven't tried in code but tried in GTM
    • Standard report Behavior > Events > Pages, group events by Content Group
    • These are not in the Site Content reports
    • These do not change the previous pageview hit Content Group
(no term)
Content Grouping is not shown on Realtime reports
Also used in Segment's Conditions Filter
content group has to be set before or at the moment pageview hit is sent
  • Page Group 1
  • Landing Page Group 1
  • Next Page Group 1
(no term)
e.g. Webpages have
  • Content Grouping:Clothing at index 1
  • a value belongs to Content Grouping:Clothing, also a Content Grouping:Men at index 2
    Shirts
    a value belongs to Content Grouping:Men
    Pants
    a value belongs to Content Grouping:Men
    (no term)
    Accessories
  • a value belongs to Content Grouping:Clothing, also a Content Grouping:Women at index 3
    Tops
    a value belongs to Content Grouping:Women
    (no term)
    Slacks
    (no term)
    Skirts & Dresses
    (no term)
    Accessories
  • Clothing, Men, and Women are options in the Primary Dimension > Content Grouping menu
    • When you select Clothing as the primary dimension, Men and Women are the dimension values in the first column of the report table
    • When you select Men as the primary dimension, Shirts, Pants, and Accessories are the dimension values in the first column of the table
    • For each dimension value (Content Group), you see behavioral metrics like Pageviews and Bounce Rate
13.4.7.3. Behavior Flow

Change the dimension (green box on the left) to Campaign to show AdWords campaign(s) behavior flow.

13.4.8. Conversions

Goals, Ecommerce, Multi-Channel Funnels, Attribution

  1. Multi-Channel Funnels

    Assisted Conversions What are the sources contribution in terms of:

    Last interaction
    the interaction that immediately precedes the conversion.
    Assist interaction
    any interaction that is on the conversion path but is not the last interaction.
    First interaction
    the first interaction on the conversion path; it's a kind of assist interaction.
    Time Lag
    how many days a conversion is made
    Top Conversion Paths
    what sources were went through before a conversion is made
    Path length
    how many interactions took before a conversion is made.

13.4.9. Add Google Search Console to a property

There will be a new section in the report called Acquisition > Search Console Need to go back to Search Console to see keywords report GA shows keyword only for non-google-logged-in users that use google to search google:ga:search console

13.4.10. Campaigns and Goals track across user session

13.4.11. Goal vs Event

http://www.metricmogul.co.uk/5-simple-differences-between-goals-and-events-in-google-analytics-and-when-to-use-them/

ga:goal

  • A goal provides additional data such as conversion rate in Acquisition reports
  • A goal will only record once per session. Newsletter sign up twice with 2 emails, only one conversion is recorded
  • An event can be anything. A goal has to be a destination, duration, pages per session or based on an event
  • A goal can be based on an event
  • Goal flow report
  • Goal cannot be shared across Views nor Properties. It has to be created in each View unless when View is duplicated
    • Goal ID 1 to 5 is Goal Set 1, ID 6 to 10 is Goal Set 2 etc.

Types of a Goal

Desitnation
the only type that is possible to track funnels
Duration
Sessions that last a specific amount of time or longer
(no term)
Pages/Screens per session
Event
multiple events in one session but event goal is counted as one
  • A conversion is counted once per session per goal
  • When Event value is empty on Goal setup page, it means it could be any value
(no term)
Smart Goal
(no term)
Destination Goal match types: begin with, equal to & RegEx

Events are good for

  • Downloaded file (shows which pages have the pdf file link)
  • Link (outbound)
  • The first 10 event hits sent to Analytics are tracked immediately, thereafter tracking is rate limited to one event hit per second
  • It's better to setup events and setup goals as the event goal type

13.4.12. Segment ga:segment

  • A Segment is to filter out the user sessions and further user that you don't want without destroying your View data
  • Segment occurs after sampling in Property for GA Standard
  • Segment by default is visible in any View across account and can only be edited by the creator. Options
    • Creator can apply/edit Segment in any View
    • Creator can apply/edit Segment
    • Collaborators and Creator can apply/edit Segment in this View
  • Admin > Property > View > Segment > Advanced > Conditions Filter
    • can use regex
  • All Users and New Users. Metrics inside each type of report will be divided into 2 so that you can compare
  • Say you want to use Segment to only include user sessions that have been to a url
    • Go to the newly created Segment, you will still see those user sessions have been to other pages
  • https://searchenginewatch.com/sew/how-to/2268458/16-secret-google-analytics-advanced-segments-worth-their-weight-in-gold
13.4.12.1. Converter segment ga:segment:converter
  • Refer to ga:goal
  • Include user sessions that have finished a goal. You can copy this system goal and make it your own e.g. finish a specific goal
  • Some stats you may find is that people who convert have lower bounce rate, higher pages/session, higher avg. session duration.
13.4.12.2. Custom segment - AdWords Search adwords:segment:search

Users Include Source/Medium = google/cpc AND Default Channel Grouping = Paid Search

13.4.13. Tracking Code - gtag.js, analytics.js ga:gtag

13.4.13.1. Basics

#+NAME gtag.js 

<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-xxx"></script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'UA-XXX');
</script>

#+NAME analytics.js 

<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');

ga('create', 'UA-XXXXX-Y', 'auto');
ga('send', 'pageview');
</script>

Old ga.js 

<script type="text/javascript">
// IMPORTANT: Remove this code snippet when upgrading to analytics.js

  var _gaq = _gaq || [];
  _gaq.push(['_setAccount', 'UA-XXXX-Y']);
  _gaq.push(['_trackPageview']);

  (function() {
    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
  })();

</script>
13.4.13.2. Custom parameters for dynamic remarketing ga:dynamic remarketing

https://support.google.com/adwords/answer/3103357 

<!-- Global Site Tag (gtag.js) - Google AdWords: 123456789 -->
<script async src="https://www.googletagmanager.com/gtag/js?id=AW-123456789"></script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'AW-123456789');
</script>

<!-- Event snippet for Example dynamic remarketing page -->
<script>
  gtag('event', 'page_view', {
  'send_to': 'AW-123456789',
  'ecomm_prodid': 'REPLACE_WITH_STRING_VALUE',
  'ecomm_pagetype': 'REPLACE_WITH_STRING_VALUE',
  'ecomm_totalvalue': 'REPLACE_WITH_STRING_VALUE'
  });
</script>

Have to use certain parameters for certain business type. Here're the list of parameters.

First, setup Custom Dimensions based on business type. https://support.google.com/analytics/answer/3455600

Next, implement tags like the above on the website

Then setup these Dynamic Attributes in GA > Admin > Property > Audience Definitions > Dynamic Attributes. This step also shares the attributes with AdWords accounts. https://support.google.com/analytics/answer/6002231

Then you can setup Dynamic Remarketing on AdWords

Maximum duration that a user can be included in a remarketing audience is 540 days.

13.4.13.3. gtag.js config analytics.js create ga:gtag:config ga:ga:create
ga:gtag:config:groups

Refer to ga:gtag:event:send_to 

// Default config sends a pageview hit. Don't send a pageview
gtag('config', 'UA-xxx', { 'send_page_view': false });

// Send a virtual pageview
gtag('config', 'GA_MEASUREMENT_ID', {'page_path': '/new-page.html'});
Cookie domain
  • Most cases set it to 'auto', it sets the _ga cookie to the highest level domain it can. e.g. if your website address is blog.example.co.uk, gtag.js will set the cookie domain to example.co.uk. If it detects that you're running a server locally (e.g. localhost), it automatically sets the cookie_domain to none
  • GA requests on websites that are subdomains and ancestor subdomains can share the same cookie
  • If cookie_domain is not set, it will be set to the current domain

  • Turn off automatic cookie domain configuration, update the config for your property to specify a value for the cookie_domain parameter 

      gtag('config', 'GA_TRACKING_ID', {
'cookie_domain': 'blog.example.co.uk'
  });

  ga('create', 'GA_TRACKING_ID', 'auto');
  // or ga('create', 'GA_TRACKING_ID', {'cookieDomain' : 'auto'});
(no term)

Cross-domain tracking allows you to see sessions from 2 sites as single session. Also called site linking. This code will append the linker parameter to any links on the page that point to the target domain 'example.com': 

gtag('config', 'GA_TRACKING_ID', {
  'linker': {
'domains': ['example.com']
  }
});
  • If the destination domain has been configured to automatically link domains, it will accept linker parameters by default

  • If the destination domain is not configured to automatically link domains, you can instruct the destination page to look for linker parameters by setting the accept_incoming property of the linker parameter to true on the destination property's config: 

      gtag('config', 'GA_TRACKING_ID', {
'linker': {
  'accept_incoming': true
}
  });

    Single snippet on all domains 

      // on example-1.com
  gtag('config', 'GA_TRACKING_ID_1', {
'linker': {
  'domains': ['example-1.com', 'example-2.com']
}
  });

  // on example-2.com
  gtag('config', 'GA_TRACKING_ID_2', {
'linker': {
  'domains': ['example-1.com', 'example-2.com']
}
  });
13.4.13.4. gtag.js set ga:gtag:set

  • Set parameters that will be associated with every subsequent event on the page 

    gtag('set', {    
'country': 'US',
'currency': 'USD',
'metric1', 1
});
13.4.13.6. Send to multiple property ids ga:gtag:event:send_to
ga('create', 'UA-XXXXX-Y', 'auto');
ga('create', 'UA-XXXXX-Z', 'auto', 'clientTracker');

ga('send', 'pageview');
ga('clientTracker.send', 'pageview');

gtag.js 

gtag('config', 'GA-TRACKING_ID-1', { 'groups': 'group1' });
gtag('config', 'GA-TRACKING_ID-2', { 'groups': 'group1' });

// Routes to 'GA-TRACKING_ID-1' and 'GA-TRACKING_ID-2'
gtag('event', 'sign_in', { 'send_to': 'group1' });

// The following two lines are equivalent:
gtag('config', 'GA-TRACKING_ID-1');
gtag('config', 'GA-TRACKING_ID-1', { 'groups': 'default' });
Full example
send to a specific property instead of groups
<script async src="https://www.googletagmanager.com/gtag/js?id=GA-TRACKING_ID-1">
</script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  // Global configs
  gtag('config', 'GA-TRACKING_ID-1');
  gtag('config', 'AW-CONVERSION_ID');
  gtag('config', 'DC-FLOODLIGHT_ID');

  // Track AdWords conversions
  gtag('event', 'conversion', {
      'send_to': 'AW-CONVERSION_ID/AbC-D_efG-h12_34-567',
      'value': 1.0,
      'currency': 'USD'
  });

  // Track Floodlight conversions
  gtag('event', 'conversion', {
    'allow_custom_scripts': true,
    'send_to': 'DC-FLOODLIGHT_ID/actions/locat304+standard'
  });

  // route ecommerce add_to_cart event to AdWords and Analytics
  gtag('event', 'add_to_cart', {
    'send_to': [
      'GA-TRACKING_ID-1',
      'AW-CONVERSION_ID'
    ],
    'items': [
      'id': 'U1234',
      'ecomm_prodid': 'U1234',
      'name': 'Argyle Funky Winklepickers',
      'list': 'Search Results',
      'category': 'Footwear',
      'quantity': 1,
      'ecomm_totalvalue': 123.45,
      'price': 123.45
    ]
  });
</script>
13.4.13.7. ga Command Queue and Object Methods

https://developers.google.com/analytics/devguides/collection/analyticsjs/tracking-snippet-reference 

<!-- Google Analytics -->
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');

ga('create', 'UA-XXXXX-Y', 'auto'); // create a tracker instance
ga('send', 'pageview');
</script>
<!-- End Google Analytics -->

ga('command_name', ...); // this is called command queue. Command can be a tracker method, a ga Object Method, or a method that is defined in a plugin

// Use a ready callback to call a ga Object Method
// ga.getAll() // wrong
ga(function(tracker) {
  var trackers = ga.getAll();
});
13.4.13.8. ga Command Queue

Add a command to the queue :: ga(command, […fields], [fieldsObject])

command
  • [trackerName.][pluginName:]methodName
  • The name of the method to be scheduled for execution. When not specifying a plugin name, this method must be one of the command methods listed below.
…fields
One or more optional convenience parameters for quickly specifying common fields.
fieldsObject
An object for specifying any remaining values not specified in any of the fields parameters.

Commands

create
create a tracker instance. same as ga Object Method create
remove
remove a tracker instance. same as ga Object Method remove
send
same as ga:tracker:send ga:ga:send
set
set a field on a tracker object. Same as ga:tracker:set
require
require an analytics.js plugin
provide
provide an analytics.js plugin and its methods for use with the ga() command queue. Add command
13.4.13.9. ga Object Method

create :: create a tracker instance getByName :: get a tracker instance by name getAll :: get all tracker instances remove :: remove a tracker instance

13.4.14. Tracker Object Methods

get
get the value of a field stored on the tracker ga:tracker:get
set
set a field value on the tracker ga:tracker:set
send
send a hit to Google Analytics. Same as ga:ga:send
13.4.14.1. set ga:ga:set

When a tracker's field is set, tracker will use that as default for all following actions/commands/methods e.g. send a hit. Use custom variable to subsitute into each action's required parameters instead of overwrite a tracker's field.

13.4.14.2. send ga:tracker:send
  1. Send pageview - Page Tracking

    Pageview fields (tracker object fields)

    title
    It's set to document.title when tracker is created
    location
    either location or the page is required. It's set to document.location when tracker is created
    • The anchor portion is ignored. Including campaign parameters in anchor but they're processed in a special way
    page
    default is not set. If set, location field gets overwritten when send method is run but tracker's location field is still not modified. ga:fieldsObject:page

    Always define tracker.page which is only used in send method. Don't modify location of document.location. Or you can use self define page variable and pass it to send method without changing tracker.page which will always be used in send method.

    For SPA, you can use a plugin to track complex pageviews :: https://github.com/googleanalytics/autotrack 

    ga('send', 'pageview');

ga('send', {
  hitType: 'pageview',
  page: location.pathname
});

ga(function(tracker) {
  // Sets the page field to "/about.html".
  tracker.set('page', '/about.html');
});

// normalize /user/USER_ID/profile, /user/USER_ID/account, /user/USER_ID/notifications with thousands of USER_ID
// to 
// /user/profile, /user/account, /user/notifications
if (document.location.pathname.indexOf('user/' + userID) > -1) {
  var page = document.location.pathname.replace('user/' + userID, 'user');
  ga('send', 'pageview', page);
}

ga(function(tracker) {
  // Sends an event hit for the tracker named "myTracker" with the
  // following category, action, and label, and sets the nonInteraction
  // field value to true.
  tracker.send('event', 'link', 'click', 'http://example.com', {
    nonInteraction: true
  });
});
  2. Funnel and Goal (send pageview) ga:goal and funnel

    Goal is defined in Admin > Account > Property > View Funnel is step to a Goal.

    Goal or funnel must use the pageview hitType.

    When path is very dynamic, you may need to send pageview with a static path in order to capture goal or funnel destination URL.

    And you may need to filter out those destination URL's in another GA View. Don't filter the current GA View.

    View the Virtual Pageview in Reporting > Real-Time > Content View the converted goal in real time Reporting > Real-Time > Conversions 

    ga('send', 'pageview', '/ga_goal/goal1/goal1_variant1.html');
ga('send', 'pageview', '/ga_goal/goal1/goal1_variant2.html');
ga('send', 'pageview', '/ga_goal/goal1/step1/goal1_variant1.html');
ga('send', 'pageview', '/ga_goal/goal1/step2/goal1_variant1.html');

// ...

gtag('config', 'GA_TRACKING_ID', {
  'page_title' : 'homepage',
  'page_path': '/ga_goal/goal1/goal1_variant1.html'
});
  3. Send Custom Event - play, call, scroll ga:send event
    • https://developers.google.com/analytics/devguides/collection/gtagjs/events
    • https://support.google.com/analytics/answer/1033068#Anatomy
    • Reporting > Behavior > Events
    • Category > Action > Label > Value
      Category
      primary dimension
      Action
      secondary dimension
      Label
      maybe third dimension
    • Syntax
      gtag.js

      gtag('event', eventName_eventAction, eventParameters);

      • gtag.js eventParameters
      <eventName>
      same as eventAction
      <category>
      the string that appears as the event category
      <action>
      the string that appears as the event action in Google Analytics Event reports
      <label>
      the string that appears as the event label
      <value>
      a non-negative integer that appears as the event value. Usually monetary value
      non_interaction
      By default, the event hit is considered an interaction hit, which means that it is included in bounce rate calculations. If the event is set to non_interaction true, then the event is not a hit
      (no term)
      analytics.js
      • ga('send', 'event', [eventCategory], [eventAction], [eventLabel], [eventValue], [fieldsObject]); or
      • ga('send', 'event', {eventCategory: '', eventAction: '', eventlabel: '', eventValue: 123, 'dimension1': '', 'dimension2': '', 'metric1': 123 }); or
      • ga('send', { hitType: 'event', eventCategory: '', eventAction: '', eventlabel: '', eventValue: 123, 'dimension1': '', 'dimension2': '', 'metric1': 123});
    gtag('event', <action>, {
  'event_category': <category>,
  'event_label': <label>,
  'value': <value>
});

gtag('event', 'play', {
  'event_category': 'Videos',
  'event_label': 'Fall Campaign'
});

$("a[href^='tel:+18881234567']").click(function(event){
    gtag('event', 'click', {
  'event_category': 'phone call',
  'event_label': '1 (888) 123-4567',
  'value': 100
    })
});

$("a[href^='tel:']").click(function(event){
  var $target=$(event.target);
  //console.log($target.attr('href'));
  gtag('event', 'click', {
    'event_category': 'Call from website',
    'event_label': $target.attr('href'),
    'value': '100'
  });
});

// does not cover tel: and mailto:
$('a[href]:not([href=""])').each(function (i) {
  if (typeof $(this).attr('href') !== 'undefined' &&
    this.hostname &&
    this.hostname.toLowerCase() !== location.hostname.toLowerCase()) {
    // outbound links
    $(this).on('click', function () {
  //console.log($(this).prop('href'));
  gtag('event', 'click', {
    'event_category': 'Outbound-Link',
    'event_label': $(this).prop('href'),
    'value': '1',
  })
    })
  }
})

gtag('event', 'video_auto_play_start', {
  'event_label': 'My promotional video',
  'event_category': 'video_auto_play',
  'non_interaction': true
});

    // scroll depth compared to the root element
var element_to_track = 'html';
var scroll_reach = 0;
var temp_scroll_reach = 0;
var scroll_reach_pixels = 0;
var temp_scroll_reach_pixels = 0;

$(window).scroll(function(){
  temp_scroll_reach = Math.floor(
    100 *
    ( $(window).scrollTop() + $(window).height() - $(element_to_track).position().top)
    / $(element_to_track).height()
  );
  temp_scroll_reach_pixels =
    Math.floor($(window).scrollTop() + $(window).height());
  if(temp_scroll_reach > scroll_reach){
    scroll_reach = temp_scroll_reach;
    scroll_reach_pixels = temp_scroll_reach_pixels;
  }
});

function sendGAreq() {
  // Old ga.js
  // _trackEvent(category, action, opt_label, opt_value, opt_noninteraction)
  //_gaq.push(['_trackEvent', 'Scroll Depth', 'Percentage', '', scroll_reach]);
  //_gaq.push(['_trackEvent', 'Scroll Depth', 'Pixels', '', scroll_reach_pixels]);
  // New analytics.js
  // ga('send', 'event', [eventCategory], [eventAction], [eventLabel], [eventValue], [fieldsObject]);
  ga('send', 'event', 'Scroll Depth', 'Percentage', '', scroll_reach);
  ga('send', 'event', 'Scroll Depth', 'Pixels', '', scroll_reach);
}

/* report only after a user exists the website */
$(window).bind('beforeunload', function() {sendGAreq();});
  4. Send dimension and metrics ga:gtag:event:custom dimension
    • Configure and send custom dimensions
    • Use chrome:google tag assistant to debug
    • create Custom Report on GA
      • Primary Dimension:Page, Secondary Dimension:Custom Dimension
      • Metric: Users
    • Wait a few minutes then create Segment using User contains Custom Dimension equal something and apply the segment to Behavior > All Pages
    • Refer to 
    ga('create', 'UA-XXXX-Y', 'auto');
ga('send', 'pageview', {
  'dimension2':  'My Custom Dimension'
});

// or
/*
ga('create', 'UA-XXXX-Y', 'auto');
// Set value for custom dimension at index 1.
ga('set', 'dimension2', 'dimensionValue');
// Send the custom dimension value with a pageview hit.
ga('send', 'pageview');
*/

// or after pageview hit is sent
ga('send', 'event', 'any_eventCategory', 'any_eventAction', {
    'dimension2': 'dimensionValue'
});

// gtag has to run 2 commands to accomplish the same

// Configures custom dimension<Index> to use the custom parameter
// 'dimension_name' for 'GA_TRACKING_ID', where <Index> is a number
// representing the index of the custom dimension.
gtag('config', 'GA_TRACKING_ID', {
  'custom_map': {'dimension<Index>': 'dimension_name'}
});
// Sends the custom dimension to Google Analytics.
gtag('event', 'any_event_name_eventAction', {'dimension_name': dimension_value});

// Maps 'dimension2' to 'age'.
gtag('config', 'GA_TRACKING_ID', {
  'custom_map': {'dimension2': 'age'}
});
// Sends an event that passes 'age' as a parameter.
gtag('event', 'age_dimension', {'age': 12});

// if multiple gtags are configured, send the event to the right UA-xxx
// Refer to ga:gtag:event:send_to
gtag('event', 'dimension_age', {
    'send_to': 'UA-xxx',
    'age': 12
});
    1. Configure and send custom metrics
      // Configures custom metric<Index> to use the custom parameter
// 'metric_name' for GA_TRACKING_ID, where <Index> is a number
// representing the index of the custom metric.

ga('send', 'event', 'category', 'action', {
  'metric18': 8000
});
ga('send', 'event', 'category', 'action', {
  'metric19': 24.99
});

// gtag has to run 2 commands to achieve the same

gtag('config', 'GA_TRACKING_ID', {
  'custom_map': {'metric<Index>': 'metric_name'}
});
// Sends the custom dimension to Google Analytics.
gtag('event', 'any_event_name', {'metric_name': metric_value});

// Maps 'metric5' to 'avg_page_load_time'.
gtag('config', 'GA_TRACKING_ID', {
  'custom_map': {'metric5': 'avg_page_load_time'}
});
// Sends an event that passes 'avg_page_load_time' as a parameter.
gtag('event', 'load_time_metric', {'avg_page_load_time': 1});
    2. Configure and send both custom dimensions and custom metrics
      gtag('config', 'GA_TRACKING_ID', {
   'custom_map': {
     'dimension2': 'age',
     'metric5': 'avg_page_load_time'
   }
});

gtag('event', 'foo', {'age': 12, 'avg_page_load_time': 1});

      // Set tracker's custom dimension so that all later hits will have this
ga('set', 'dimension5', 'custom data');

// set both
ga('set', {
  'dimension5': 'custom dimension data',
  'metric5': 'custom metric data'
});

13.4.15. fieldsObject ga:fieldsObject

https://developers.google.com/analytics/devguides/collection/analyticsjs/field-reference

13.4.15.1. page ga:fieldsObject:page

text begins with /

13.4.15.3. hitCallback
var form = document.getElementById('signup-form');

// Adds a listener for the "submit" event.
form.addEventListener('submit', function(event) {

  // Prevents the browser from submitting the form
  // and thus unloading the current page.
  event.preventDefault();

  // Sends the event to Google Analytics and
  // resubmits the form once the hit is done.
  ga('send', 'event', 'Signup Form', 'submit', {
    hitCallback: function() {
  form.submit();
    }
  });

  // or 
  ga('send', 'event', 'Signup Form', 'submit', {
    transport: 'beacon'
  });

  form.submit();
});

13.4.17. Analytics Solutions Gallery

  • https://analytics.google.com/analytics/gallery
  • A gallery may contain Custom Reports, Dashboard, Segments, etc.
  • Imported gallery will show in Admin > View > Share Assets
  • Dashboards, Custom Reports, Segments, Goals, and Custom Attribution Models can be shared in the Solutions Gallery

13.4.18. Email tracking, Measurement Protocol

The Google Analytics Measurement Protocol allows developers to make HTTP requests to send raw user interaction data directly to Google Analytics servers
https://developers.google.com/analytics/devguides/collection/protocol/v1/email

13.4.19. Site Search

  • Admin > choose a View > View Settings
    Query Parameter
    up to 5 e.g. s,q for a.ca/?s=abc&q=xyz
    Strip query parameters out of URL
    only strip the parameters set up as above and not the other parameters
    Site search categories
    e.g. a.ca/?s=abc&cat=xyz where cat is the parameter to setup, which inidicates to search in what categories
  • Behavior > Site Search
    Usage
    Sessions that use Site Search vs not
    Search Terms

13.4.20. Custom Alerts

  • Admin > choose a View > Custom Alerts
    Period
    Day, Week, Month

13.4.21. Google Analytis Core Reporting API

13.4.22. Remarketing with Analytics ga:remarketing

13.4.25. Transfer a property

Move Property
https://support.google.com/analytics/answer/6370521
(no term)
Have Manage Users and Edit permissions on both the source and destination accounts (GA account level)
(no term)
Property ID does not change
(no term)
Permissions
Replace existing property and view permissions with permissions of the destination account
the property and its views will inherit perms from the destination account
Keep existing property and view permissions
Source account perms will be copied to destination account. Users who have account-level access in the source account will have property-level access in the destination account
(no term)
Can't move a property if

  • The source account and the destination account are not in the same google:marketing platform organization

    • Administration > Click on 3 dots on source org, move accounts from this org to another

    • Required permissions

      • Source org: Org admin and Billing admin
      • Destination org: Billing admin
      • If the source org can't add you as an Org admin or Billing admin, then you should consider to unlink the GA account from source org and then link it to destination org after
  • The service level for the property is set to 360 and the account link to the organization has not been verified
  • The property is linked to Google Ad Manager (only GA 360 can link GAM)
    • Unlink and relink after the account transfer

13.4.26. Track 404 pages

Behavior > Site Content > All Pages > add Page Title as secondary dimension, add filter Page Title containing Page Not Found

13.4.27. TS: Too Many Self-referrals

Reporting > Acquisition > All Traffic > Referrals, you will find the website's url is the source And the percentage is high compared to other Referral Sources. 3 reasons:

  1. Users sessions idle for 30 minutes and become active. This should be < 10% of traffic
  2. Cross domain (need to bring it down to 0 self-referral)
  3. A lot of traffic and sessions are created by spammers to a landing page

(most likely homepage with url parameters) with a fake referral path as the website.

You cannot use View Filter to filter out referral domain…

You can also create a segment without affecting the data. Conditions > Filter: Sessions, Exclude > Source / Medium > contains > OHG.com / referral

Ghosts :: send request to GA directly without visiting your site. Use random UA-ID and destination url.

  • Identify the source domain of referral traffic Report > Acquisition > Source/Medium, change primary dimension to Source and secondary dimension to Hostname. If Hostname is not your webiste, then the Source is the ghost's referral domain. Full Referrer dimension shows referrer's domain and path.
  • Create a filter under Property > View to only include Hostname that is your website
    • Custom > Include > Hostname > yourmaindomain\.com|anotheruseddomain\.com

Don't do this as it will turn all referral traffic coming from that domain as direct traffic!

  • Go to Admin > Account > Property > Tracking Info > Referral Exclusion List > add domain

abc.com also matches www.abc.com because domain name CONTAINS

  • Usually you add your own domain to Referral Exclusion List

https://moz.com/blog/stop-ghost-spam-in-google-analytics-with-one-filter More detail instructions :: https://carloseo.com/removing-google-analytics-spam/

13.4.29. Checklist

  • Audience > Audiences > enable Demographics and Interest Reports
  • Admin > Property Settings >
    • Enable Demographics and Interest Reports ga:audience:interests
    • In-Page Analytics, Use enhanced link attribution
    • Search Console
  • Admin > Adwords Linking
  • Admin > Audience Definitions > Audiences
    • For a property, ga:remarketing
    • Set Audience Destinations to the corresponding AdWords account
    • Define your audience and give a name
    • In AdWords campaign > Audience > Edit > under Targeting, Remarketing to select the list from GA
  • Admin > Properties > View
    Create 3 views
    All website data, external traffic only, test view
    (no term)
    For external traffice only view, enable Bot Filtering - Exclude all hits from known bots and spiders
    (no term)
    Duplicate External Traffic Only view as Test View

13.5. Google Optimize google:optimize

Optimize vs Optimize 360
https://www.google.com/analytics/optimize/compare/
  • Free version can only have one container in one account (container is like property in GA). Container ID is like GTM-XXXXX
(no term)
A cookie is set when a user enters a variant and the user will continue to see that variant until both _ga and _gaexp cookies are removed
(no term)
An experiment runs on all pages unless you have targeting setup and you can select a page to run the editor to configure the variants
(no term)
Link to GA can compare GA metrics in variants

13.6. GA Dev Tools: Query Explorer

13.7. GA Dev Tools: Campaign URL Builder google:campaign-url-builder

13.8. Google Search Console google:search console

  • Add a new property for https if the website is upgraded from http. Stats are separated
  • Google recommends creating 2 properties per root domain per schema (http/https). domainroot.com and www.domainroot.com
  • Domain Property is introduced so you should create a domain property on top of all URL properties (http|https, non-www|www) for a website
  • A URL Property will be automatically created if a GA account is given access. You still need to manually link Search Console from GA
  • https://support.google.com/webmasters/
  • Initial HTML is crawled and indexed, and later on JavaScript is run (rendering) and a second-wave of indexing happens
    • Second-wave could take several days
    • js:serviceworker, local & session storage, IndexedDB, Web SQL, Cookies, Cache API

13.8.1. Performance - Search Results Report

Total Stats
https://support.google.com/webmasters/answer/7042828
  • Total clicks
  • Total Impressions
  • number of impressions per click
  • avg of the topmost position
(no term)
Queries, Pages, Countries, Devices
impressions
how many times your site appears on search result page
click
how many times people click on your website on search result page
position
(no term)
More than 16 months of stats are available
(no term)
Refer to wp:plugin:wordpress-seo

13.8.2. Index - Coverage

  • Error

    • Submitted URL seems to be a Soft 404

      • Live Test on each URL > View Tested Page
      More Info
      HTTP Response should be 200
      • Page resources
      • JavaScript console messages
      Screenshot
      if content is rendered and not too many assets are blocked

13.8.3. Web Tools

13.8.4. Remove URL from Google Index

  • This is to permanently remove a url from Google Search
  • If a page is recently deleted, returning 404 and doing the following is the right way to do
    • Yoast adds meta tag to 404 pages
    • Google Search Console will stop showing the 404 after about a month
    • But you should pay attention if the url is in sitemap but returns 404 (shown as 404 error on Search Console)
  • Make the URL return 404 or 410 status
  • Don't block access to that URL using robots.txt
  • Return meta tag <meta name="robots" content="noindex, nofollow">
  • Return X-Robots-Tag HTTP header
  • Go to Google Search Console and Fetch as Google to manually tell Google update index for that URL
  • Submit a removal request
    For sites you own on Google Search Console
    https://www.google.com/webmasters/tools/url-removal
    For sites that you don't own
    https://www.google.com/webmasters/tools/removals
  • To check if the URL is removed 
<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

$apacheURL = (strlen($_SERVER['REQUEST_URI'])) ?
  substr($_SERVER['REQUEST_URI'],1) : $_SERVER['REQUEST_URI'];
$noIndexNoFollow = array(
  "~^subscriber-services/trucking-renewal(.*)~i",
  "~^lookup-subscription(.*)~i",
  "~^users/(.*)~i",
  "~^user/(.*)~i",
);

if (!empty($apacheURL)) {
  foreach ($noIndexNoFollow as $key => $value) {
    $matches = array();
    if (preg_match($value, $apacheURL, $matches)) {
  header("X-Robots-Tag: noindex, nofollow", true);
    }
  }
}

13.9. Google Trends

  • Compare search terms in different Google Search platforms and see their related topics and related queries
    Compare 2 keywords
    hiking boots, hiking shoes
    • Plural vs sigular spelling
    (no term)
    Traffic change over year
  • A B
    • must contain both words
    • order doesn't matter
    • other words can be added before, after or in-between
    • Spelling is exact
  • "A B"
    • exact phrase inside double quotation marks
    • other words can only be added before or after
  • A + B
    • A or B
    • center + centre + centere
      • include alternative spellings. Consider each version of a word a different search
  • A - B
    • contain A but exclude B
  • Special characters
    • Apostrophe, single quotes and parentheses will be ignored
    • women's tennis world ranking you get result for womens tennis world ranking
  • Terms vs Topics
    • Term is literal match
      • banana matches "banana sandwich"
      • "banana sandwich" matches "banana for lunch" and "peanut butter sandwich"
  • Topic is a group of terms that share the same concept in any language
    • London matches "capital of the UK"
    • also matches "Londres" which is London in Spanish

13.10. Google Shopping Insights

Search products (Google defines them). For US only. Nation > region > city level. By age, household income, sex, device.

13.11. Google Ads - Google AdWords, Google Keyword Planner, Google Video Advertising

13.11.1. Keyword Planner google:ads:keyword-planner

  • Keyword Ideas
    • Hiking, hike, backpacking, backpack
  • Get keyword's search volume, avg cpc and recommendation on other relelvant keywords
  • Targeting
    • Geo
    • Language
    • Search network

13.11.2. Google Ads - AdWords

AdWords requires 1+ hour/week and AdWords Express requires 15 min/week. Express only has text ads on Search and Google Maps and doesn't require to have a website. Academy for Ads http://school4seo.com/category/google-adwords-advance-search/ https://academy.exceedlms.com/student/catalog/list?category_ids=53

13.11.2.1. Campaign Goals

Build awareness

Targeting
Demographics, Affinity/custom affinity audiences, Managed Placements
Measure
Views, Impressions, Unique users, Awareness lift, Ad recall lift
Solutions
six-second, non-skippable, in-stream video ads, Masthead

Influence consideration

Targeting
In-market audiences, Audience keywords, Similar audiences, Audience Keyword targeting, Content keyword targeting, Topic targeting
Measure
View-through rate, Watch time, Favorability lift, Consideration lift, Brand interest lift
Solutions
TrueView in-stream ads, Gmail

Drive action

Targeting
Display remarketing, Dynamic remarketing
Measure
clicks, calls, sign-ups, app installs, purchase intent lift, sales
Solutions
Gmail

Grow loyalty

Targeting
Display remarketing

qq

13.11.2.2. Keywords Match Type adwords:keywords match types
Default broad match
hiking tour northern california
  • match with queries with any order and any additional words
  • match with synonyms and mispellings
  • partial match is allowed "best hiking" query matches hiking tour northern california
Modified broad match
+ sign includes broad match but excludes synonyms
  • +gel batteries must include gel or its synonym, batteries may not present
  • gel +batteries must include batteries or its different forms, gen may not present
Phrase match with double quotes
any queries match "hiking tour northern california" in any order with additional words
Exact match and only these words
[hiking tours northern california]
Negative match
-dog, -wallpaper, -screensavers, -pictures to ignore queries that have this negative word
(no term)
google:search operators are ignored
(no term)
https://support.google.com/adwords/answer/7476658
(no term)
https://support.google.com/adwords/answer/2472708?hl=en
13.11.2.3. Negative keywords

Negative broad match :: ad won't show if the search contains all negative keyword terms, even if the order is different. running shoes Phrase match :: ad won't show if the search contains the exact keyword terms in the same order. Exact match :: ad won't show if the search contains the exact keyword terms, in the same order, without extra words.

https://komarketing.com/blog/200-plus-negative-keywords-to-consider-for-b2b-ppc/ hawaii career careers "new balance sale" "sale in new brunswick"

13.11.2.4. Campaigns, Ad groups
Campaign
It's usually for a type of product you sell. If budget or location targeting is changed, create another campaign
Setting
frequency capping, start/end dates, budget, goal, locations, language, bidding strategy, devices, content exclusions, IP exclusions, dynamic ads feeds, ad rotation, campaign URL options
Frequency capping
level can be set ad, ad group and this campaign
Drafts & Experiments
A draft inherits all settings from a campaign and almost any ads and ad groups can be changed. After a draft is made, it can be applied to the original campaign or creat an experiment to test how changes perform against your original campaign
Limitation
some features and reports that are not available for drafts, and some features aren't supported by experiments
Experiments
while multiple drafts for a given campaign, only one draft can run as an experiment at a time
  • Specify how long to run and how much original campaign's traffic (and budget) to use for the experiment
  • Later, an experiment can be applied to the original campaign or convert the experiment to a new campaign
Ad group
It holds one or more ads which target a shared set of keywords (theme) or a set cost-per-click (CPC) bid
  • To show ads that are relevant to the searches of people you're trying to reach, bundle related ads together with related keywords into an ad group. That way, all of your related ads can be shown to customers searching for similar things
  • For each ad group, use keywords related to that theme. Consider also having your ads mention at least one of your keywords in its headline
  • bidding, ad rotation, ad group targeting, landing pages, keywords, placements, demographics, audiences, narrow targeting, observations, automated targeting
    Ad Group Targeting
    audiences, demographics, keywords, placements, additional observations, automated targeting
Ad
URL, creative, Search Ad (title, URL, description)

Organize account with ad groups

Linking to someone's AdWords account lets that person advertise your locations with location extensions. Linking to someone's Merchant Center account lets that person advertise your products with local inventory ads.

Dynamic Search Ads

  • A campaign can enable DSA and the default ad group (1st ad group when a new Search campaign is created) is a dynamic ad group
  • Standard Ad Group can be added as well.
  • Only Dynamic Search Ads can be added to this dynamic ad group. For each ad, provide just a description field
  • Dynamic Ads Targets are created to categorize webpages of your destination website (domain). And you can further select categories to target the Dynamic Search Ads created in this dynamic ad group
13.11.2.5. Placement

Locations on the Display Network where your ad can appear. Examples include relevant websites and apps that partner with Google to show ads.

13.11.2.6. Quality Score, Ad Rank, Ad Position
  • Quality Score
    Relevancy
    someone searches "snowboard rentals" and your ad uses keyphrase "Snowboard rentals Tahoe"
    Click-through rate (CTR)
    clicks over views
    Account history
    AdWords account
    Landing page relevancy
    content match what people search (bounce rate)
  • adwords:ad rank
    • Ad Rank = Quality Score x Maximum bid x ad formats.
13.11.2.7. Bidding Strategy - Media Cost Model

Campaign goals decide on which media cost model you should use.

Use manual bidding first and after the campaign is already getting at least 15 conversions and 15 clicks per month, automated bidding can help:

Maximize clicks
Good for advertisers just starting off with online marketing who are most interested in getting customers to their website.
Ad revenue
Target return-on-ad-spend (tROAS). Good for advertisers who know the exact value of each conversion.
Ad conversions
Good for advertisers looking to drive conversions. "auction-time bidding"
Enhanced cost per click (ECPC)
described later
Maximize conversions
max number of conversions. May use up all daily budget for one conversion
Target cost per acquisition (tCPA)
max number of conversions while maintaining avg CPA. Good for multiple campaigns with various CPA. Some bids might cost higher than the corresponding target you set but some are lower in order to make up the avg cost is within your set target. CPA is the average amount you pay for a conversion.

Cost per Click (CPC) is determined by max bid + Quality Score + Ad Rank and compare those against your competitors. Drive traffic.

Set Max. CPC to = your profit x commission for Google x your conversion rate

$100 x 0.3 x 1% = $.30 1% conversion rate is based on 1000 views of the page and 10 people buy

Manual CPC
each keyword or Ad Group would have the same bid
Automatic CPC
there're some bid strategies under this category. Max CPC cannot be set
Enhanced CPC (ECPC)
relies on Google’s own historical data to help you predict where and when to adjust bids you set manually in order to get more conversions. Its CPC is constrained by Max CPC you set
  • e.g. if a campaign’s performance looks promising it will automatically raise bids to ‘capture’ more results (for less money). Similarly, it will also drop bids if necessary to help you save on wasted ad spend if performance starts to slide.
  • it's better to turn it off initially and let AdWords collect data and further turn it on
  • individual CPC might exceed Max CPC you set, but avg CPC is below Max CPC

CPC, CPA and CPM are 3 bidding strategies but each one can be further optimized based on bid modifiers which are rules that raise or lower bids based on:

  • Geographic locations
  • although more traffic from mobile but conversion rate is higher on desktop
  • Dayparting

3 phases

Brand awareness
Let the world know about you. Requires larger budget due to the longer path to conversion and the scale at which you try to reach people.
Influence consideration
Encourage customers to explore you.
Driving action or sale
lower funnel usually requires less budget.
  1. CPA - boost sales

    Cost Per Action (or conversion)

  2. CPM, vCPM - brand awareness
    • Applies only to the Display Network (along with remarketing campaigns, too). Here you pay a cost (like a few cents or dollars) per one thousand impressions
    • viewable cost per thousand impressions. Cost a bit more than CPM.
  3. CPV - boost video views

    cost per view for video or click on it.

  4. Bid Simulators

    https://support.google.com/adwords/answer/2470105

    Click Campaigns, Ad groups, or Keywords. If you're in Campaigns, the icon is in the "Daily budgets" column. If you're in Ad groups or Keywords, the icon is in the "Default max. CPC" column.

    Some reasons why bid simulators do not show

    • Search and Display campaigns that use Manual CPC, Enhanced cost-per-click, and Target CPA bid strategies.
    • campaign has reached or nearly reached its daily budget at least once in the last 7 days
    • a campaign, keyword, ad group, or product group was recently added, or didn't receive many impressions or clicks in the last 7 days.
    • campaign uses shared budgets
  5. First-page bid estimate

    For CPC, AdWords can show estimates for showing your ad on:

    • First page
    • at the top of first page
    • First position

    AdWords > Keywords > Add columns > Attributes > Est. first page bid, Est. top of page bid, Est. first pos. bid

  6. Bid Adjustments
    • Bid adjustment is available in adwords:targeting
    • Decrease a bid by 100% is to completely exclude the ad from a targeting setting
  7. Daily budget

    Divide by 30.4 that is avg number of days per month.

    Delivery method

    • Standard
    • Accelerated
13.11.2.8. Networks, Campaign Types, Ad Formats

Search network is likely to bring more traffic to your site. Display network is for driving awareness e.g. promote for a new website or something that has a new concept

All campaign types

  1. Search Network

    Google Search, Shopping, Maps and Google Play for websites and Google apps. Also Google Search Parnters. Search Parnters :: non-Google websites, as well as YouTube and other Google sites. CTR for ads on search partner sites doesn't impact Quality Score on google.com. It is included by default. To remove it AdWords account > Settings > select campaign > Networks > uncheck Include Google search partners

    Ad formats

    text ads
    Search Network and Search Partners
    ad extension
    adwords:ad extension
    (no term)
    dynamic search ads
    shopping ads
    Google Search, Shopping, Search Partners
    image ads
    Display Network, Search Partners but not Google Search Network
    video ads
    Display Network, Search Parnter Networks but not Google Search Network
    app promotion ads
    Search Network, Display Network
    call-only ads
    Search Network
  2. Display Network

    Shows ads on other websites, other apps, YouTube, Gmail, Blogger. New or changes can take 12-24 hours to apply.

    Display Network can target audiences by interests, topics, placements. adwords:targeting

    Ad formats

    Responsive ads
    responsive can fit in all available sizes and can transform into text or image ads.
    • Landscape: ideally sized at 1200 x 628
    • Square: ideally sized at 1200 x 1200
    • A short headline (25 characters or fewer) which appears in tight ad spaces. It may or may not appear with a description.
    • A long headline (90 characters or fewer) which appears instead of the short headline in larger ads. In some cases, it may be shortened with ellipses. It also may or may not appear with a description.
    • A description which may appear with the headline and invite users to take action. It also may be shortened with ellipses.
    Image ads
    Creat as many sizes and formats as possible in order to reach on Display Network.
    (no term)
    rich media ads
    Engagement ads
    image and video ads on YouTube and across the Display Network
    Gmail ads
    expandable ads on the top tabs of people's inboxes. Can't use affinity, remarketing or remarking-based audiences. May interests and domain targeting (receive emails from).
    Video ads
    remarketing on people who view your YouTube videos or channel as they browse Display Network.
    (no term)
    app promotion ads
    (no term)
    adwords:ad extension
  3. Shopping
  4. Video
    TrueView in-stream ads
    YouTube and across Display Network sites, games, or apps.
    TrueView video discovery ads
    YouTube.
    Bumper ads
    6 seconds or less.
  5. Universal App

    Downloads of your app

13.11.2.9. Targeting options adwords:targeting
Keywords
try to include between 5 and 20 keywords per ad group
Audience
Show ads to people likely to be interested in these keywords and also on webpages, apps, and videos related to these keywords
Content
Only show ads on webpages, apps, and videos related to these keywords e.g. webpage's concepts.
(no term)
Audience
Interests and habits (Affinity and custom affinity)

types of websites not webpages the audience have visited. This is for running TV ad

  • Interests entered as keywords
  • URL
  • places
  • apps
    • Custom Affinity is only for Display Campaign
Life evnets
engage with viewers on YouTube and Gmail
Researching or planning (In-market and custom intent)
likely buyers
(no term)
Remarketing
Similar Audience
better to create another campaign. Not all custom audience list can have a similar audience
(no term)
Audience can be added on a campaign or an ad group
Demographics
age, gender, income
Topics
topics a webpage has e.g. central theme
Placements
specific websites your ad will show on. May also be specific YouTube videos, channel.
(no term)
Ad schedule
Locations
majority of consumers want ads customized to their city, zip code, or immediate surroundings
(no term)
Settings
13.11.2.10. Ad Group Automated Targeting
  • Only for Display Network, under Ad Group > Settings > Automated targeting
  • default
  • broader match. e.g. if your keyword is “pens,” conservative targeting may extend to “felt-tip pens" and “ballpoint pens,” but aggressive targeting might show your ads in contexts related to “whiteboard markers” or “mechanical pencils”—if there’s data to suggest that those keywords will lead to conversions. Plus, automatic targeting works for remarketing. Aggressive targeting is available for all Display Network campaigns with at least 15 conversions per month. It costs a little bit more than Conservative.
  • When to use automatic targeting
    • Find more customers
    • Identify the best targeting to reach your most likely customers
    • Increase reach without increasing bids or cost per customer
13.11.2.11. Ad Formats in networks

https://support.google.com/adwords/answer/1722124

  1. text ad

    adwords:text ad Goal :: specific, actionable, and relevant. Best practice. Required *

    *Headline
    consider including at leat one keyword. 2 fields up to 30 characters each. Fields are combined with hyphen.
    *URL
    Path field appear as www.example.com/Indoor-Plants but actual is different.
    *Description
    80 characters

    Some good phrases are:

    • Shop now, Buy now, Get an instant quote online, See pricing
13.11.2.12. Ad extension
  1. Basics adwords:ad extension

    Extension appears with ads on the Search Network. Some extensions might also appear with ads on the Display Network.

    Extensions are free to use. List of extensions.

    Factors determining when they show:

    • Your Ad Rank, which combines your bid, the quality of your ad and landing page, Ad Rank thresholds, context of the person’s search, and expected impact of extensions and other ad formats. AdWords requires a minimum Ad Rank (factoring in your extensions) before showing extensions. So, you may need to increase your bid or your ad quality (or both) in order for your extensions to show.
    • The position of your ad on the Google search results page. There’s a limited amount of space available above Google search results for ad extensions, and ads in higher positions get the first opportunity to show extensions. Ads in lower positions generally won’t have more extensions than ads in higher positions. Plus, the AdWords system generally won’t allow ads in lower positions to get more incremental clicks from extensions than the incremental clicks they’d get from moving up to a higher position. To show ads in higher positions, generally you need to increase your ad quality, bid, or both.
    • Other extensions you’ve enabled. In each auction, we'll generally show your highest performing and most useful combination of eligible extensions and formats. You will not be able to get a combination of extensions which gives more expected click-through-rate (CTR) than the expected CTR of a higher ad position.

    Both ad formats and ad extension could help your ad's Ad Rank!

    Extensions can be set on campaign level.

  2. Location extension

    Works on both Search and Display Network including Google Maps. https://support.google.com/adwords/answer/7040605

    Encourage people to visit your business by showing your location, a call button, and a link to your business details page—which can include your hours, photos of your business, and directions to get there.

  3. Call extension

    Has some limitation on Display Network.

  4. Message

    Encourage people to send you text messages from your ad.

  5. Price

    Showcase your services or product categories with their prices, so that people can browse your products right from your ad.

  6. Promotion

    Highlight specific sales and promotions across your ads (e.g., 30% off rose bouquets)

  7. App

    Available globally for Android and iOS mobile devices, including tablets.

  8. Sitelink

    Show six to eight additional links to different pages on her site. e.g. store hours, location details, and popular bouquets

  9. callout

    Callouts can improve your text ads by promoting unique offers to shoppers, like free shipping or 24-hour customer service. When customers see your ads, they get detailed information about your business, products, and services.

    2 to 6 callouts show in addition to the text of your ad. Ads with callout extensions can show at the top and bottom of Google search results. When callout extensions show, they appear below your ad copy.

    Free Shipping · 24-7 Customer Service · Price Matching

    Callout text is limited to 25 characters in most languages, or 12 characters in double-width languages (like Chinese, Japanese, and Korean).

  10. Structured snippet

    Highlight specific aspects of your products and services with structured snippets extensions. Structured snippets show beneath your text ad in the form of a header (ex: "Destinations") and list of values (ex: "Hawaii, Costa Rica, South Africa").

    Services: Tech Support, E-Waste Recycling, Computer Repair

    List of headers Amenities Brands Courses Degree programs Destinations Featured hotels Insurance coverage Models Neighborhoods Service catalog Shows Styles Types

    It can show up to 2 headers at a time, while ads that show on mobile and tablet devices will only show one header. AdWords algorithmically decides the best header or combination of headers to show, so it’s best to add as many headers as possible that are relevant to your business.

  11. Universal extensions

    Sitelink, Callout and Structured Snippets are called universal extensions that fit all business objectives.

13.11.2.14. ValueTrack URL parameters

These parameters can be used in Final URL and Tracking Template. Here's an example of Tracking Template 

{lpurl}?matchtype={matchtype}&device={device}

All parameters https://support.google.com/adwords/answer/6305348

{keyword}
For the Search Network: the keyword from your account that matches the search query, unless you are using a Dynamic Search ad, which returns a blank value. For the Display Network: the keyword from your account that matches the content.

Parallel tracking should be enabled :: AdWords account > All campaigns > Settings > Account Settings > Tracking > Parallel tracking

13.11.2.15. Supported Ad Sizes

Square and rectangle 200 × 200 Small square 240 × 400 Vertical rectangle 250 × 250 Square 250 × 360 Triple widescreen 300 × 250 Inline rectangle 336 × 280 Large rectangle 580 × 400 Netboard

Leaderboard 468 × 60 Banner 728 × 90 Leaderboard 930 × 180 Top banner 970 × 90 Large leaderboard 970 × 250 Billboard 980 × 120 Panorama

Skyscraper 120 × 600 Skyscraper 160 × 600 Wide skyscraper 300 × 600 Half-page 300 × 1050 Portrait

Mobile 300 × 50 Mobile banner 320 × 50 Mobile banner 320 × 100 Large mobile banner

Most popular for responsive ad (covers 95% in Display Network) :: 300x250, 728x90, 1600x600, 320x50, 300x600

IAB ad portfolio 2017

Google AdSense top performing ad sizes

13.11.2.16. Conversion tracking
Conversion tracking with Google Analytics
https://support.google.com/google-ads/answer/2375435
  • google:ads:link GA
  • google:ads:auto-tagging
  • If GA has linked to the AdWords account and goals are defined on GA, AdWords conversion tag may not be necessary to put on the website
  • Tools > Measurement > Conversions > New > Import Google Analytics
    • New stats are imported when Import is clicked. Conversion data can be seen on AdWords within 2 days or 9 hours behind GA shows goal conversion stats. Historical data before the import won't be included
    • If GA goal newly created or a GA goal is newly imported to Google Ads, it may take about 30 minutes for it to appear on Google Ads
Conversion tracking using Google Ads conversion tag (on websites, mobile apps, call tracking)
https://support.google.com/adwords/answer/1722054

  • Advantage about Google Ads conversion tracking

    View-through conversion
    Measures how many visitors saw ads on Google Display Network and YouTube Video but did not interact/click but they later complete a conversion on your site
    • Google Ads conversion tracking is required. Cannot simply setup goals on Google Analytics
  • https://support.google.com/adwords/answer/1722021
(no term)
Conversion tracking data (columns) seen on Google Ads once conversion tracking is set up

Use global site tag on website same as remarketing 

<script async
src="https://www.googletagmanager.com/gtag/js?id=AW-CONVERSION_ID"></script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());
  gtag('config', 'AW-CONVERSION_ID'); // or you just add this to your existing GA global site tag

  // If you want to manually send an event to mark a conversion:
  // gtag('event', 'conversion', {'send_to': 'AW-CONVERSION_ID/conversion-unique-specific-hash-id'});
</script>
  1. Phone adwords:conversion:phone

    Call Extension or call-only ad, enable Call Reporting and you have to change "count conversions as" to a new Conversion tracking instead of the default "Calls from ads"

    https://support.google.com/adwords/answer/6095882

    Measurement > Conversions > Add new Phone calls - Calls from ads using call extensions or call-only ads

    After that, go back to your existing call extension or call-only ad or when you create them from now on Select the conversion action you just created as Count conversion as

    1. Calls to a phone number on your website
      <script async
src="https://www.googletagmanager.com/gtag/js?id=GA_TRACKING_ID"></script>

<script>
  window.dataLayer = window.dataLayer || [ ] ;
  function gtag(){dataLayer.push(arguments);}
  gtag( 'js', new Date () ) ;

  gtag( 'config', 'GA_TRACKING_ID');
  gtag( 'config', 'AW-CONVERSION_ID');

</script>

<script>
  gtag('config', 'AW-CONVERSION_ID/CONVERSION_LABEL', {
    'phone_conversion_number': '1-650-555-5555'
  });
</script>

      If you select "Don't enter a number" 

      gtag('config', 'AW-CONVERSION_ID/CONVERSION_LABEL', {
  'phone_conversion_number': '1-650-555-5555',
  'phone_conversion_callback':function(formatted_number, mobile_number) {
  // formatted_number: number to display, in the same format as the 
  //     number passed to 'phone_conversion_number'. 
  //     (in this case, '1-650-555-5555')
  // mobile_number: number formatted for use in a clickable link
  //      with tel:-URI (in this case, '+16505555555')
  var e = document.getElementById("number");
  e.innerHTML = "";
  // e.href = "tel:" + mobile_number;
  e.appendChild(document.createTextNode(formatted_number));
  };,
  'phone_conversion_options':timeout=20;cache=false'
});

      Adding this will get a telephone number and replace the contents of all spans of the given class 

      phone_conversion_css_class: 'number'

      <span class="number">1-800-123-4567</span>
  2. Attribution Modeling

    It is only available for clicks on Search Network and Shopping ads on Google.com. Which ad click makes the final conversion?

    Data-driven :: only available for search campaigns with 15k+ Google.com search clicks and 600 conversions per conversion type over 30 days. Position-based :: 40% of credit to both first and last clicked ads and corresponding keywords, remaining 20% spread out across the other clicks Time decay :: More credit to clicks that happened closer in time to the conversion Linear :: Credit for the conversion is shared equally across all clicks on the path

13.11.2.17. Remarketing Audience, Customer Match
  1. Rules and Templates
    Rules
    https://support.google.com/adwords/answer/6297497
    • Standard parameters include URL, Referrer (e.g. www.google.com coming from search or anothersite.com), parameters sent through the event snippets and attributes from Google Merchant Center
    • Path and URL query parameters can be targetted. But not #
    Template
    https://support.google.com/adwords/answer/6297549
    • Visitors of a page
      • Add multiple conditions and if any condition met, then add to the list
    • Visitors of a page who also visited another page
      • To create a list of people who visited pages A, B, and C, first create a list of people who visited pages A and B using this template. Then, using the "Visitors of a page" template, create a list of people who visited page C. Then, create a custom combination using these two lists.
      • Only on Display Network
    • Visitors of page who did not visit another page
      • When you create a list using this template, you narrow your audience. To be on your list, visitors need to visit the page defined in the first rule AND not visit any of the pages defined in the second rule. This is what makes this template different from the "Visitors of a page" template. When you create a list using the "Visitors of a page" template, you can add different conditions to the rule, but people who match ANY of the conditions (as opposed to ALL of the conditions) will be added to the remarketing list.
      • Only on Display Network
    • Visitors of a page during specific dates
      • When creating lists with specific dates, lists' membership duration may also determine when people are removed from the list. Visitors are added to the list if they've visited the pages on the selected dates, and will stay on the list for the amount of time specified by the membership duration.
    • Visitors of a page with a specific tag
      • Awhile ago, an airline added a remarketing tag to sections of their website about popular routes. Creating a rule based on the URL for those sections would be too complex using rules. The airline could create a remarketing list of people who visited sections of the website about the popular route by using the "Visitors of a page with a specific tag" template, and selecting the tag that was implemented a while ago on those pages.
      • When you use the "Visitors of a page with a specific tag" template, you can select existing AdWords conversion tracking tags as well. Selecting a conversion tracking tag can be helpful if you haven't been able to add the remarketing tag to the conversion page but still want to reach or exclude this audience.
  2. Custom Combination List

    https://support.google.com/adwords/answer/2549111

  3. Customer Match

    It's for Search, YouTube and Gmail Networks.

    • Upload a list as an Audience List to match Google accounts.
    • Similar audience targeting based on your Customer Match audiences is available for YouTube and Gmail.
13.11.2.18. Reports
  1. System Reports

    Search terms :: When you use broad-match keywords (the default setting), your ads can appear when someone searches for a variation of your keyword, like a similar phrase or related word. To see a list of searches that have triggered your ad, use the Search terms report. You can use this report to identify relevant terms that are driving traffic to your website, and then add them as new keywords. Or, if any of the keywords are irrelevant to your business, you can add them as negative keywords so they won't trigger your ads.

    Paid and organic report :: how people got to you — comparing Google’s free organic search results to your paid AdWords ads. Learn the ways customers are looking for products and services like yours and update your own keyword list or create new ad groups to directly target them. In order to access this report you’ll need to link Google Webmaster Tools to Google AdWords. AdWords > Linked Accounts.

    User locations :: physical locations Geographic :: Location of interest and physical locations.

    Landing page experience :: located under Keywords > a keyword > column Status to review Ad Rank

    Wrench icon > Measurement > Search Attribution :: Once you've set up conversion tracking, you'll have access to a handy set of reports about your conversions. Attribution reports show you the paths customers take to complete a conversion, and attribute the conversion to different ads, clicks, and factors along the way.

    Return on investment ROI is calculated with this formula: (Revenue - Cost) / Cost. Example: If your ad resulted in $1,200 of sales for a product that cost $600 to make, and your advertising cost was $200, then your ROI is [$1,200 - ($600 + $200)] / ($600 + $200) = 50% ROI.

    Segments For any reports, you can click on 3 bars next to filter to create segment

    • Network
    • Device
    • Time
    • Click type
    • Conversions
    • Top vs. Other
    • Keyword text

    Auction Insights :: how well you did in auctions compared to other advertisers that compete with you? Available to Search Network and Shopping. Keywords > Auction Insights

    Reach Metrics (only in Campaigns):

    Unique users
    same user but across multiple devices
    (no term)
    Unique cookies
    Unique viewers
    same as cookie but for Video ads only
    Frequency
    an estimate of the average number of times a unique cookie was exposed to your ad over a given time period.

    Low CTR but a lot of impressions :: If the goal is max sales, try to lower the Max CPC.

    High conversion rate and low CPC :: increase Max CPC for this keyword.

  2. Custom Reports

    Custom reports are made on Account level. Filter to a specifc campaign if necessary.

    e.g. Row:Campaign Columns:All stats include Phone impressions, Phone calls and phone-through rate (Call details).

    Individual row and column can be further filtered.

    If you want to break down to individual goals, use GA.

    GA does not show AdWords phone conversion.

    1. Breakdown Conversions inside Google Ads

      Breakdown conversions #1 (User Location & Ad Groups)

      Rows
      Ad group, Most specific location target, Region, Conversion action, Device, Day
      Columns
      Conversions

      Breakdown conversions #2 (Search Term & Ads)

      Rows
      Ad, Search term, Conversion action, Device, Day
      Columns
      Conversions

      Breakdown conversions #3 (Hours of Day & Ad Groups)

      Rows
      Ad Group, Hour of Day, Conversion Action, Device, Day
      Columns
      Conversions

      Breakdown conversions #4 (Extensions & Ad Groups)

      Rows
      Extension, Ad Group, Day, Conversion Action, Device, Day
      Columns
      Conversions
13.11.2.19. Autotagging google:ads:auto-tagging
  • Always enable autotagging in AdWords
    • Previous view > Gear icon > Account Settings > Preferences > Tracking > Auto-tagging
    • Auto-tagging is turned off by default. Settings on the left page menu > Account Settings along the top > Auto-tagging. On if "Tag the URL that people click through from my ad" is checked
  • https://support.google.com/analytics/answer/1733663

If the final URL contains any of the following, manual tagging is used:

  • utm_source, utm_medium, utm_campaign, utm_content, utm_term

When auto-tagging is used, gclid=abc parameter is appended to the url. Auto-tagging only works with Google Analytics e.g. other 3rd party tools which only use UTM parameters.

You can add UTM parameters and with auto-tagging on, UTM parameters and gclid will be in the URL. You can also let GA take the UTM parameters you specifiy instead of the values that gclid can bring.

13.11.2.20. Automated Rules
13.11.2.21. Third party impression tracking

Submit a form to tell Google you want a 3rd party impression/view/skip/DCM Enhanced YouTube Pixel tracking.

Only available to Display and Video campaigns.

13.11.2.22. Accounts google:ads:account
Either users in Normal Account or users in Manager Account
will have access to all campaigns/settings of the Normal Account
To link a Normal Account to a Manager Account
so that the Manager Account can manage the Normal Account. On the Manager Account, on the left Accounts > Management > + and Link existing accounts, copy and paste the Normal Account's Customer ID, Send Request. Go to the Normal Account > Tools & Settings > Account access > Managers > approve. You can set the Manager Account as Administrative owner while on Normal Account, that way the Manager Account can create/approve/revoke access for any others
Normal account
also called managed account or individual account
1 Customer ID
123-123-1234. Click on icon ? in the top right corner
Multiplue Users
who may modify the whole parent account
Multiple Manager Accounts (Managers)
who can modify the whole parent account
To link a Manager Account to this account
go to the Manager Account and add this account's Customer ID
Campaigns
1 Payments Account ID
  • Each Google Account has a Payment Profile. A Normal Account can have its own Payments Profile
    • For Google Account, add users to Payments Profile with permissions when Payments Profile Account Type is Business
    • For Normal Account, add contacts to receive email notifications only
  • https://support.google.com/google-ads/answer/7058401
  • You can set it up on the Manager Account and link it to this account, then go to Billing of this account to add the Manager Account's Payments Profile
  • The 1st step requires to contact Google's online specialist
Manager account
One Google Account (email) can only see 20 Normal Accounts. You need a Manager Account with users to access more Normal Accounts
Create one
use a different email that is not used for any Normal Account or Manager Account. https://ads.google.com/home/tools/manager-accounts/
1 Customer ID
123-123-1234. Click on icon ? in the top right corner
Multiple Users
Multiple Manager Accounts (Managers)
Views
Overview
Recommendations
Accounts
Campaigns
Change History
Partners program
13.11.2.23. Link a Google Analytics Property and Google Ads accounts google:ads:link GA
  • Linking a GA property to Google Ads account can help you analyze customer activity on your website after an ad click or impression
  • The same Google Account should have Edit permission for the Analytics property and Admin access for the AdWords manager account
  • Sign in to Google Ads account
    • Click the tools icon in the upper right corner, under Setup, click Linked accounts
    • Under Google Analytics, click Details
    • Search the GA property you want to link e.g. UA-xxx and click Link. If the property is not in the list, check if you have GA Edit permissions

    • After linking, you can how many views of that GA proerpty are linked. Click on the Views to select which views you want to link. 2 settings for each view

      Link
      make Google Ads click and cost data available in GA and GA goals and transactions available from import into Google Ads
      Import site metrics
      import site engagement metrics from GA. It's used to show site engagement metrics in GA reporting columns on Google Ads
      Need to add GA data (reporting columns) to Google Ads reports
      https://support.google.com/google-ads/answer/2617364

13.11.4. Google Video Advertising

https://support.google.com/displayspecs AdWords :: https://support.google.com/adwords/answer/2375464

Google Preferred :: Goolgle serve your video ads on YouTube videos that are the top 5% content across highly popular channels, playlists or video collections. Videos are divided into several categories.

Desktop and Mobile Video Mastheads :: https://support.google.com/displayspecs/answer/6244544. YouTube homepage 100% share of voice, target country and device and device.

DoubleClick Bid Manager :: programmatic buying platform

13.11.4.1. TrueView in-stream ads
  1. TrueView in-stream

    In-stream ads play before or during another video from a YouTube partner. Viewers see five seconds of your video and then have the choice to keep watching or skip it.

    You pay when a viewer watches for at least 30 seconds or to the end of the video (whichever is shorter) or clicks on a card or other elements of your in-stream creative.

    Interactive Features you can add:

    • Call-to-action overlays
    • End screens
    • Cards
    • Auto end screens

    https://support.google.com/youtube/answer/150471?hl=en

  2. TrueView video discovery

    Video discovery ads appear alongside other YouTube videos, in YouTube search pages, or on websites on the Google Display Network that match your target audience.

    You pay only when a viewer chooses to watch your video by clicking on the ad.

13.11.4.2. Measure & Solutions

Awareness

  • Views, Impressions, Unique users, Awareness lift, Ad recall lift
  • six-second, non-skippable, in-stream video ads
  • Masthead

Consideration

  • View-through rate, Watch time, Favorability lift, Consideration lift, Brand interest lift
  • TrueView in-stream ads

Action

  • Clicks, Calls, Sign-ups, App installs, Purchase intent lift, Sales
13.11.4.3. Video Remarketing

Remarketing lists :: YouTube related actions or Google Search Display behaviors. Views, likes, commenting, sharing and subscribing, as well as visiting channels, videos or even mastheads

These lists can be used in existing, new and even Search and Display campaigns.

In AdWords :: Linked Accounts > Youtube channel > you own or someone else owns

YouTube users :: viewed any video from a channel, viewed certain videos, viewed any video (as an ad) from a channel, viewed certain videos as ads, subscribed to a channel, visited a channel page, liked any video from a channel, added any video from a channel to a playlist, commented on any video from a channel, shared any video from a channel

Features on AdWords after linking YouTube channel to an AdWords account

View counts and calls-to-action
view ad completion rates of videos, create CTA overlays on videos from linked channels.
Remarketing
create lists based on viewers' past interactions on linked channels
Engagement
view earned actions metrics from videos and video ads from linked channels.

Multiple AdWords accounts can be linked to a YouTube channel.

Metrics

View rate
% of people who choose to watch video ads after seeing the video ad's thumbnail.
Avg. CPV
cost per view
(no term)
CTR
(no term)
Cost
Earned actions
happen when a viewer watches a video ad and then takes a related action on YouTube.
Earned views
watch subsequent videos on the channel or Watch pages
Earned subscribers
happen when a viewer subscribes to your channel.
YouTube Analytics
  • Audience retention report (e.g. relative audience retention for a video compared to the YouTube average for similar videos)
  • calls-to-action and other interactive elements, # of shares, where/when you gain or loose subscribers

Contact Google representative to conduct a Brand Lift survey to show metrics in business meaning :: lifts in awareness, ad recall, brand interest, consideration, brand favorability, purchase intent. You will need to provide several of your competitors in order to conduct the survey.

13.11.4.4. YouTube Advertising

https://www.youtube.com/yt/advertise/ Learn how to make a video https://creatoracademy.youtube.com/page/welcome

13.11.5. Google Rich Media Gallery

Templates, Learn, Formats/Ad Format Gallery

13.12. Google Tag Manager google:gtm

13.12.1. Code

Container > Admin > Install Google Ad Manager 

<head>
  <!-- as high in the <head> as possible -->
  <!-- If JS libraries are used for GTM, then put this after the libraries are loaded -->
  <!--
  Define dataLayer for GTM to define Variables for triggers or tags
  dataLayer will persist as long as the visitor remains on the current page
    -->
  <script>
    dataLayer = [{
    'pageCategory': 'signup',
    'visitorType': 'high-value'
    }];
    <!-- a diffrent custom parameter can also be used in GTM -->
    myNewName = [];
  </script>

  <!-- Google Tag Manager -->
  <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':
    new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
    j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
    'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);
    })(window,document,'script','dataLayer','GTM-CONTAINER-ID');</script>
  <!-- End Google Tag Manager -->

  <!-- Change to use custom parameter myNewName instead of default dataLayer -->
  <!--
  <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':
    new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
    j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
    'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);
    })(window,document,'script','myNewName','GTM-CONTAINER-ID');</script>
  -->

  <!-- dataLayer is defined after GTM is loaded -->
</head>
<body>
  <!-- immediately after the opening <body> -->
  <!-- Google Tag Manager (noscript) -->
  <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-CONTAINER-ID"
            height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>
  <!-- End Google Tag Manager (noscript) -->

  <!-- After GTM is loaded, should use push() API to modify dataLayer -->
  <a href="#" name="button1" onclick="dataLayer.push({'event': 'button1-click'});" >Button 1</a>

</body>

After that, click SUBMIT to publish the change

13.12.2. Tag

  • Tag Sequence
    • Choose another tag to run before or after the current tag
    • Hence, a tag might not need a trigger
  • Types

    • Custom HTML you can paste custom JavaScript 

        <script>
console.log('local hour: {{Local Browser Time-hour}}');
  </script>
      • Enable Support document.write if it doesn't have external sources

13.12.3. Variables

  • Variables can be referred to in Tag and Trigger as value using double curly brackets e.g. {{Page URL}}
  • Built-in variables without any tags. These are not all built-in variables:
    All built-in variables
    https://support.google.com/tagmanager/topic/7182737
    (no term)
    Event
    (no term)
    Page Hostname
    (no term)
    Page Path
    (no term)
    Page URL
    Click URL
    after a link is clicked
  • Variables come with a certain tag type
    • TransactionTotal = {insert cart checkout total here}
    • GA Custom dimensions
  • Google Analytics Settings
    • Used to config multiple GA tags
    • Some events do not use the Settings' content group for the events
  • https://support.google.com/tagmanager/topic/9125128
    • Constant Variable
    • dataLayer.push({"Data Layer Name": "value"}) google:gtm:user-defined variable:data layer
      Version 1 :: dataLayer.push( "a.b.c": "value")
      interpreted as { "a.b.c": "value"}
      Version 2 :: dataLayer.push({"a.b.c": "value"})
      interpreted as {a: {b: {c: "value"}}}
    • Put e.g. document.title as Global Variable Name and this GTM JavaScript Variable is the same as that variable

    • these vars can be available as soon as Page View. Refer to trigger 

      function() {
return new Date().getHours();
}

13.12.4. Trigger

  • Trigger controls under which condition a Tag is used/inserted. A Tag always requires at least one trigger
  • Types
    Page View
    2 types
    Page View
    fire immediately when the browser begins to load a page
    DOM Ready
    Pageview-based tags that interact with the DOM to populate variables should use this
    Window Loaded
    fire when page has fully loaded
    (no term)
    Click
    Just Links
    The following example fire an event to track outbound clicked link
    • Create a Tag to be fired after GTM event is triggered
    (no term)
    Universal Analytics
    Type
    Event
    Category
    link clicks
    Action
    outbound
    Label
    {{Click URL}}
    Tracking ID
    UA-XXX
    • Create a trigger for the Tag
    Trigger Type
    Click - Just Links
    (no term)
    Click URL does not contain yourwebsitedomain.com
    (no term)
    This trigger fires on All Link Clicks
    Custom Event
    google:gtm:trigger:custom event
    Trigger Group
    Only fire after all of the selected triggers have fired at leaswt once

13.12.5. Preview

Click Preview on GTM and normally refresh the webpage and you will get a preview pane just on your browser

13.12.6. UC: Custom Dimension WordPress Post Terms google:gtm:custom dimension

13.12.6.1. On website
<head>
    <script>
     (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({
     'gtm.start':  new Date().getTime(),event:'gtm.js'});
     var f=d.getElementsByTagName(s)[0],
         j=d.createElement(s),
         dl=l!='dataLayer'?'&l='+l:'';
     j.async=true;
     j.src='//www.googletagmanager.com/gtm.js?id='+i+dl;
     f.parentNode.insertBefore(j,f);
     })(window,document,'script','dataLayer','GTM-XXXX');
    </script>
    <!-- End Google Tag Manager -->
    <script>
     if (typeof dataLayer !== 'undefined') {
     // dataLayer is defined after GTM
     <?php
     $terms = $terms_id = [];
     if (is_single()) {
         global $post;
         $tax = get_taxonomies();
         $terms = wp_get_post_terms($post->ID, $tax);

         if ( !is_wp_error($terms) && !empty($terms)) {
         foreach ($terms as $term) {
             $terms_id[] = $term->term_taxonomy_id.'-'.$term->slug;
         }
         $terms_id[]='';
         array_unshift($terms_id, '');
         }
         //echo '//'.implode( ',', $terms_id );
     }
     if (!empty($terms_id)) : ?>
     dataLayer.push({
         'event': 'ga-custom-dimension-post-terms',
         'ga-custom-dimension-post-terms': '<?php echo implode( ',', $terms_id ); ?>'});
     <?php endif; ?>
     }
    </script>
</head>
13.12.6.2. Google Analytics

Create Custom Dimension Page Terms, hit type Hit. Say Index number is 1

13.12.6.3. Google Tag Manager
  • Create Trigger My Trigger Name
    Type
    Custom Event
    Event name
    ga-custom-dimension-post-terms (name does not matter but has to match the code on webpage)
    (no term)
    All Custom Events
    (no term)
    The code on website will trigger this event along with a set User-Defined Variable as described below
  • Create User-Defined Variable
    Type
    Data Layer Variable
    Name
    ga-custom-dimension-post-terms (name does not matter but has to match the code on webpage)
    (no term)
    Version 2
    Default value
    blank (which is empty string)
  • Create Tag
    Type
    Google Analytics - Universal Analytics
    Track Type
    Event (submit a GA Event)
    Category
    does not matter. Use ga-custom-dimension-post-terms
    Action
    does not matter. Use ga-custom-dimension-post-terms
    Label
    may not matter.. Use custom variable ga-custom-dimension-post-terms
    Enable overriding settings in this tag
    More Settings becomes available
    Custom Dimension
    Index:1, Dimension Value:custom variable ga-custom-dimension-post-terms
    Tracking ID
    GA-XXX
    Advanced Settings
    Tag firing options
    Once per event
    Triggering
    choose the Custom Event ga-custom-dimension-post-terms previously set

13.13. Google Surveys google:surveys

13.14. Google Marketing Platform google:marketing platform

  • https://marketingplatform.google.com
  • Create an Organization and link products to an organization. e.g.
    • Link a GA account with properties

    • Move a linked product account e.g. GA account to a different organization

      • Required permissions
      • Source organization: Org admin and Billing admin
      • Destination organization: Billing admin
      • In Source Organization on Marketing Platform, click 3 dots icon in the org card and click Move accounts from this org to another
        • Choose a destination org and choose the accounts you want to move
  • Show integrations of the supported products with Google Ads and Search Console
  • User Management
    • Organization administrators
      • Org Admin
      • User Admin
      • Billing Admin
    • Users
      • After linking product accounts to the org, any users who have access to that product account are automatically added to the org with the default role of User
      • Any Org User who also has Manage Users permissions for other product accounts can add the product accounts to the org
    • User Groups
    • Policies that govern who has access to the organization
  • Supported products
    • google:ga
    • have to individually shared on Data Studio. Email address should be a Google Account
    • google:optimize
    • google:surveys
    • google:gtm

    • Enterprise includes 2 other DSP products

      DSP
      Demand Side Platform provides advertisers features for buying ad placements online in real time
      • Audience targeting
      • Advertiser Campaign Management products that can manage ads across display, mobile, social, search and video advertising channels
      • Integrate with ad exchanges which serve as a marketplace for ad inventory
      • publishers use SSPs to sell ad inventory in ad exchanges
      • Some DSPs can integrate directly with SSPs to foster direct buys
        ATD
        Adgency Trading Desk
      • ATDs hire software developers, account managers and data analysts who work for the agency's advertising clients. ATDs can use multiple DSPs. Advertisers using the services of an ATD don't get direct access to the available media inventory
        • Display & Video 360
        • Search Ads 360

13.16. Website Keywords

Page title
65 characters. A list of keywords (<=3) with separators can be used. Order matters.
(no term)
Only one <h1>
(no term)
How to find keywords to invest
Keyword Research Tools (KRT)
find what keywords are within the search engine ranking
Google Search Console
see which keywords/queries end up to your site
Micromoments
people try to ask small questions for a small duration but several times throughout a period
  • I want to know, go, do and buy
  • This leads to Buying Cycle
    • Early and general
    • Research and investigative
    • Conversion
    • Post-conversion

13.17. Site Audit Tools - check js errors, GA tagging, etc.

13.17.1. ObservePoint

13.18. SEO, Search PPC Strategy Tools and Services

13.18.1. SpyFu.com

Free and no registration is required.

  • organic keywords, monthly SE organic clicks
  • Google organic clicks vs paid clicks
  • paid keywords, monthly PPC clicks, monthly Google Ads budget
  • top organic/paid keywords and cost and CPC per paid keyword
  • Google Ads history including real ads and corresponding landing pages
  • Inbound Links

13.18.2. SERanking.com

Free and no registration is required. Same as SpyFu but it breaks down by country and different SE.

13.18.3. Moz.com

  • Moz uses Jumpshot's clickstream data combined with Google AdWords
13.18.3.1. Free tools
  1. Keyword Explorer
    • Some other competitors are
    • 5k queries per month
    • Metrics
      Difficulty
      invest on high volume keyword with lower difficulty
      Low
      20-35
      Middle
      36-50
      Tough
      51-65
      Very difficult
      66-80
    • After search on Google, Google tells number of results. The more results means more competition
    • Keyword Analysis is about to analyze how people use keywords to search content that you provide (e.g. you provide California Vacations)
      Groups
      food? activity?
      (no term)
      Concepts
      Phrase pattern
      people search vacation ideas more than vacation packages
      (no term)
      Prefix
      (no term)
      Suffix
      (no term)
      Synonyms
      Plurals
      people use singular when they want to be specific and plurals when they want general info
  2. Open Site Explorer (similar to Link Explorer)
13.18.3.2. Pro tools
  1. On-Page Grader
    • Specify a page url and a target keyword, it returns with recommendation
  2. Link Explorer
    • Page Authority, Domain Authority, Linking Domains, Inbound Links, Ranking Keywords, etc.
    • Analyze all websites including yours
    • Alternatives
      • SEMrush
      • linkpopularity.com
      • checkyourlinkpopularity.com
      • majestic.com
      • Link-Assistant.com
      • RavenTools.com
      • SEO-SpyGlass.com
      • aHrefs.com

13.18.4. Hype Stat - HypeStat.com

13.18.5. MediaRadar

Estimate unique visitors

13.18.6. SimilarWeb.com

  • Similar Web allows you to see GA like web traffic stats. Most of the time estimates are overestimated

13.18.7. Answer the Public AnswerthePublic.com

13.18.8. Keywords Everywhere

https://keywordseverywhere.com/

13.20. Remarketing vs Retargeting

Remarketing is an umbrella term which covers offline, phone, email, social media and websites. Retargeting is a subset of remarketing

Retargeting:

  • people have been on your site
  • people have been on a competitor site

13.21. Competitive Marketing

  • https://www.wordstream.com/blog/ws/2016/05/16/competitive-advertising
    • Using Facebook Ad, you can target people with interests in our competitor’s company name
    • Use YouTube campaigns, you can target people with searches for our competitor’s videos on YouTube
    • Use Gmail campaigns, you can target people with interests in our competitor’s company name and its products. E.g. target people who are interested in Sephora
    • Use Display campaigns, you can target people with visits to AND interests in a specific domain. This is new but it is different from remarketing audience. But it’s powerful enough!
    • Use Twitter Ads, you can download a competitor’s followers and setup Twitter Ads to target those followers

13.23. Rich snippet, JSON LD google:json-ld

13.23.1. keywords and syntax tokens

13.23.1.1. @id

The node with @id can be referrenced either internally or externally on internet. The hash indicates Organization object not a page 

{
"@context": "http://schema.org",
"@id": "https://www.apple.com/#organization"
"@type": "Organization",
}
13.23.1.2. @graph

Contain an array of nodes that are linked using @id

13.23.2. Product

<script type="application/ld+json">
{
  "@context" : "http://schema.org",
  "@type" : "Product",
  "name" : "Product 1",
  "image" : "http://mysite.com/uploads/1.jpg"
}
</script>

Multiple products 

<script type="application/ld+json">
[ {
  "@context" : "http://schema.org",
  "@type" : "Product",
  "name" : "Product 1",
  "image" : "http://mysite.com/uploads/1.jpg"
}, {
  "@context" : "http://schema.org",
  "@type" : "Product",
  "name" : "Product 2",
  "image" : "http://mysite.com/uploads/2.jpg"
} ]
</script>

13.23.3. Organization, Website google:json-ld:organization

{
  "@context": "http:\/\/schema.org",
  "@type": "Organization",
  "url": "https:\/\/mywebsite.com\/",
  "sameAs": [],
  "@id": "#organization",
  "name": "My Organization Name",
  "logo": "http:\/\/mywebsite.com\/wp-content\/uploads\/2018\/02\/logo.jpg",
  "contactPoint": [
    { "@type": "ContactPoint",
  "telephone": "+1-401-555-1212",
  "contactType": "customer service"
    }
  ]
}

13.23.5. LocalBusiness

https://developers.google.com/search/docs/data-types/local-business https://www.chrisains.com/seo/local-business-schema-mark-via-json-ld/

It should have single address. 

<script type="application/ld+json">
{
    "@context" : "http://schema.org",
    "@type" : "LocalBusiness",
    "name" : "Your Business Name",
    "url" : "http://www.your-domain.co.uk",
    "logo": "http://www.your-domain.co.uk/images/your-logo.png",
    "image": "http://www.your-domain.co.uk/images/your-image.png",
    "address": {
    "@type" : "PostalAddress",
    "streetAddress": "1 The High Street",
    "addressLocality": "Colchester",
    "addressRegion": "Essex",
    "addressCountry": "US",
    "postalCode": "CO1 1AB",
    "telephone" : "01234 567 891"
    },
    "openingHours": [ "Mo-Fr 09:00-18:00", "Sa 10:00-16:00" ],
    "priceRange": "$$$",
    "geo": {
  "@type": "GeoCoordinates",
  "latitude": "40.75",
  "longitude": "73.98"
    },
    "hasmap" : "https://www.google.co.uk/maps/place/Buckingham+Palace/@51.501364,-0.1440787,17z/data=!3m1!4b1!4m5!3m4!1s0x48760520cd5b5eb5:0xa26abf514d902a7!8m2!3d51.501364!4d-0.14189",
    "sameAs" : [ "https://www.facebook.com/chrisains",
    "https://twitter.com/chrisains",
    "https://plus.google.com/+ChrisAinsworth",
    "https://www.youtube.com/user/chrisains1982" ] 
}
</script>

Multiple LocalBusiness's. This method Google shows no organization… 

// Include google:json-ld:organization

// LocalBusiness 1
{
  ...,
  @id: "https://example.com/url-first-location",
  ...,
  "parentOrganization": {
    "@type": "Organization",
    "@id": "Same as organization:@id",
    "name": "Same as organization:name" // Google requires a name for object
  }
}

13.23.6. OpeningHoursSpecification

"openingHoursSpecification": [
  {
    "@type": "OpeningHoursSpecification",
    "dayOfWeek": [
  "Monday",
  "Tuesday",
  "Wednesday",
  "Thursday",
  "Friday"
    ],
    "opens": "09:00",
    "closes": "21:00"
  },
  {
    "@type": "OpeningHoursSpecification",
    "dayOfWeek": [
  "Saturday",
  "Sunday"
    ],
    "opens": "10:00",
    "closes": "23:00"
  }
]

Open 6pm on Saturday and close on 3am on Sunday 

"openingHoursSpecification": {
  "@type": "OpeningHoursSpecification",
  "dayOfWeek": "Saturday",
  "opens": "18:00",
  "closes": "03:00"
}

Open all day on Saturday and close all day on Sunday 

"openingHoursSpecification": [
  {
    "@type": "OpeningHoursSpecification",
    "dayOfWeek": "Saturday",
    "opens": "00:00",
    "closes": "23:59"
  },
  {
    "@type": "OpeningHoursSpecification",
    "dayOfWeek": "Sunday",
    "opens": "00:00",
    "closes": "00:00"
  }
]

Close for Winter 

"openingHoursSpecification": {
  "@type": "OpeningHoursSpecification",
  "opens": "00:00",
  "closes": "00:00",
  "validFrom": "2015-12-23",
  "validThrough": "2016-01-05"
}

13.23.7. CreativeWork

https://schema.org/CreativeWork

More specific types e.g. Article, Blog, Book, Comment, Game, Map, Movie, Website etc.

13.23.8. VideoObject

{
    "@context": "https://schema.org",
    "@type": "VideoObject",
    "name": "Video Name",
    "description": "...",
    "thumbnailUrl": "http://...jpg",
    "uploadDate": "2018-07-09T13:00:00.000Z",
    "duratin": "T1M33S",
    "contentUrl": "http://....mp4"
}

13.24. robots.txt

  • https://developers.google.com/search/reference/robots_txt
  • On Pantheon, a custom robots.txt can be specified but it only works for Live with custom domain. For Pantheon domains, Pantheon always serves its own version
  • Valid domains
    • Not valid for other subdomains, protocols or port numbers
    • Valid for all files in all subdirectories on the same host, protocol and port number
    • It has to be in the root folder
    • http://example.com/robots.txt">valid for http://example.com/ and http://example.com/*. Not valid for https://example.com, http://example.com:8181/
  • Sitemap
    • It's require only if you are not the owner and can't submit sitemaps on Google Search Console
    • Can specify only one sitemap
    • Use full URL with http|https
  • Default
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# Disallow: /wp-content/plugins/
# Disallow: /readme.html
# Disallow: /refer/

# Allow a useragent
# User-agent: PowerMapper
# Allow: /

# start-of-group
# group-member
# non-group

# All group-member records after a start-of-group record up to the next start-of-group record are treated as a group of records
# Only user-agent can start a group
# Multiple consecutive start-of-group lines will follow the group-member records following the final start-of-group line
# Any group-member records without a preceding start-of-group record are ignored
# Only disallow and allow can be a group-member record
# sitemap is a non-group record and should be placed at the end
# All non-group records are valid independently of all groups

Sitemap: http://www.codeinwp.com/post-sitemap.xml

# Pantheon non-Live and non-patheonsite.io domain robots.txt
# Pantheon's documentation on robots.txt: https://pantheon.io/docs/bots-and-indexing/
User-agent: *
Disallow: /

User-agent: RavenCrawler
User-agent: rogerbot
User-agent: dotbot
User-agent: SemrushBot
User-agent: SemrushBot-SA
User-agent: PowerMapper
Allow: /

13.25. Disavow.txt

Submit to https://www.google.com/webmasters/tools/disavow-links 

# block a whole domain
domain:spamwebsite.com
# block individual page url
https://www.spamwebsite.com/paid-links/mysite.html

13.26. Design & User Experience

48 pixels width and height for buttons and they should be 32 pixels width and height separated apart.

13.27. hreflang html:hreflang

  • https://support.google.com/webmasters/answer/189077
  • Google Search Console looks for hreflang to determine language and location. Say if you want to target a generic domain like .com to a specific country/language, you should use hreflang and set in Google Search Console
  • For more than one language, use multilingual plugins like wp:plugin:polylang to insert hreflang
  • 3 ways to implement
    • Inside <head>
    • HTTP response header

    • To indicate a different language version of a URL (e.g. PDF) 

      Link: <http://es.example.com/document.pdf>; rel="alternate"; hreflang="es", <http://en.example.com/document.pdf>; rel="alternate"; hreflang="en", <http://de.example.com/document.pdf>; rel="alternate"; hreflang="de"
    • Sitemap
    • ISO 639 and 3166 Country Codes

13.27.1. Inside <head>

<!-- General -->
<link rel="alternate" hreflang="en-gb" href="http://en-gb.example.com/page.html" />
<link rel="alternate" hreflang="en-us" href="http://en-us.example.com/page.html" />
<link rel="alternate" hreflang="en"    href="http://en.example.com/page.html" />
<link rel="alternate" hreflang="de"    href="http://de.example.com/page.html" />

<!-- Canada -->
<link rel="alternate" hreflang="fr-ca"    href="http://www.example.com/fr-ca/about" />
<link rel="alternate" hreflang="en-ca"    href="http://www.example.com/en-ca/about" />
<link rel="alternate" hreflang="en"    href="http://www.example.com/en/about" />

<link rel="alternate" hreflang="x-default" href="http://www.example.com/" />
<!-- The reserved value hreflang="x-default" is used when no other language/region matches the user's browser setting. This value is optional, but recommended, as a way for you to control the page when no languages match. A good use is to target your site's homepage where there is a clickable map that enables the user to select their country.

// wp
-->
<?php global $wp;
  $current_url = home_url(add_query_arg(array(), $wp->request)); 
  // echo esc_url($current_url);
?>
<link rel="alternate" hreflang="<?php echo get_bloginfo('language'); ?>" href="http://es.example.com/" />

13.27.2. Sitemap

<url>
  <loc>http://www.domain.com/about/</loc>

  <xhtml:link
  rel="alternate"
  hreflang="en-ca"
  href="http://www.domain.com/en-ca/about/"
  />
  <xhtml:link
  rel="alternate"
  hreflang="fr-ca"
  href="http://www.domain.com/fr-ca/about/"
  />
  <xhtml:link
  rel="alternate"
  hreflang="en"
  href="http://www.domain.com/en/about/"
  />
</url>

13.28. Google News

  • https://news.google.com
  • https://news.google.com/publisher
    Docs
    https://support.google.com/news/publisher-center
    (no term)

    Set up content source e.g. website as publisher

    Request inclusion in News Index
    Not necessary but recommended
    <?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9">
  <url>
    <loc>http://www.example.org/business/article55.html</loc>
    <news:news>
  <news:publication>
    <news:name>The Example Times</news:name>
    <news:language>en</news:language>
  </news:publication>
  <news:publication_date>2008-12-23</news:publication_date>
  <news:title>Companies A, B in Merger Talks</news:title>
  <!-- Optional -->
  <news:keywords>k1, k2, k3</news:keywords>
    </news:news>
  </url>
  <!-- <url> ... </url>
Other news:* elements refer to https://www.google.com/schemas/sitemap-news/0.9/sitemap-news.xsd
-->
</urlset>
    (no term)
    Assign Labels to Source
    • Create sections for a Source

14. UX and Design

14.1. Optimal Workshop

Usability Testing

Tree testing
test how well readers can find specific content through navigation
Card sorting
how readers would sort or categorize content sections
First click testing
what is the first click if readers want specific content?

Ontario Digital Service - Service Design Playbook

  • Ethnographic research
  • Personas
  • Journey maps
  • Service blueprints
  • Affinity mapping
  • Inclusive Design

14.2. Mobile Design

Easy navigation

  • CTA button in front and centre position
  • Secondary CTA in menu or below the fold
  • Keep menu short, always have a button to go home

Easy search

  • Search should be one of the first things at the top users see
  • Display high relevant search results first
  • Provide search filters, preview of # of results, don't show zero results
  • Provide previous search
  • Ask questions to prefill filters

Easy conversions

  • Don't put gate too early which asks for commitment before users actually use the site
  • Prefill fields
  • Provide CTA for users don't want to fill up complex forms
  • Finish converting on another device on another day e.g. save and email jobs to apply for later

Easy forms

  • Provider features e.g. number pads, date picker
  • Users like tapping toggles instead of text enter or dropdown menu
  • Error-check entries immediately
  • Multi-part form with progress bar on top

Easy on mobile

  • Make product images expandable
  • Provide comparison features
  • Be responsive with visual feedback after significant actions
  • Ask for permissions in context e.g. show overlay over a map to ask for permissions for geolocation

14.3. Google Material Design

Material Design Guidelines :: https://material.io/design/

Material Design Components (MDC)

Codelabs :: https://codelabs.developers.google.com/?cat=Design

14.4. Style Guide

http://styleguides.io/

14.5. Pick colors

15. Linux

15.1. Explain Shell

  • linux:man
  • show the executable path for the current environment
  • only searches certain directories but return path and also man page path

15.2. System Version, Distro

uname -a all info
uname -r Kernal release
   
cat /etc/*release Linux distro
cat /etc/issue  
cat /proc/version  
lsb_release -a Ubuntu only
   
  • uname -a
    • Linux your-server-name 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
      -n
      network node host name your-server-name
      -r
      kernal release 4.4.0-116-generic
      -v
      kernal version #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018
      -m
      machine hardware name x86_64
      -p
      processor type x86_64
      -i
      hardware platform x86_64
      -o
      operating system GNU/Linux
  • Ubuntu only linux:lsb_release

    • lsb_release -a All info: 

      No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.3 LTS
Release:        16.04
Codename:       xenial
    • Ubuntu release e.g. bionic
    • Output can be changed by apt-get dist-upgrade. After running it, it shows 16.04.04 using kernal 4.4.0-116-generic
    • When Ubuntu is first installed as 16.04.0, kernal generic 4.4 is installed and kernal 4.4.0-x will always be used when 16.04.0 is upgraded to 16.04.x
    • It looks like if generic kernal version is installed at the first place, it will stay at that version until special action to replace the generic kernal
  • Linux distro
    • cat /etc/redhat-release
  • cat /etc/issue
  • cat /proc/version

15.2.1. Debian

cat /etc/debian_version cat /etc/os-release

15.2.2. Upgrade

  • sudo apt-get upgrade (-s for dry run)

Install newest versions of all packages currently installed on the system from the sources /etc/apt/sources.list Packages currently installed with new versions will be retrieved and upgraded. No packages are removed or added. e.g. upgrade Firefox use this. New versions of currently installed packages that cannot be upgraded without changing the install status of another package will be left at their current version. An update must be performed first so that apt-get knows that new versions of packages are available.

  • sudo apt-get -s dist-upgrade (-s for dry run)

In addition to performing the function of upgrade, also intelligently handles changing dependencies with new versions of packages; apt-get has a "smart" conflict resolution system, and it will attempt to upgrade the most important packages at the expense of less important ones if necessary. So, dist-upgrade command may remove some packages. The /etc/apt/sources.list file contains a list of locations from which to retrieve desired package files. See also apt_preferences(5) for a mechanism for overriding the general settings for individual packages.

By applying all package upgrades, the latest stable kernel will be pulled in.

  • sudo apt-get full-upgrade (Ubuntu 14.04+)

full-upgrade performs the function of upgrade but may also remove installed packages if that is required in order to resolve a package conflict.

CentOS :: sudo yum update Fedora :: sudo dnf update

sudo reboot

15.2.3. Ubuntu upgrade to a newer release

sudo do-release-upgrade

update-manager-core is needed sudo apt-get install update-manager-core

Usually do these 2 first sudo apt-get update, upgrade and dist-grade and then do-release-upgrade

https://www.digitalocean.com/community/tutorials/how-to-upgrade-to-ubuntu-16-04-lts

15.2.4. Kernal Bootloader

By default, the highest versioned kernel will boot.

To see the boot order: grep 'menuentry ' $(find /boot -name "grub.cfg") | cut -f 2 -d "'" | nl -v 0

Default 0 will boot. To change, edit /etc/default/grub 

. . .
GRUB_DEFAULT=saved
GRUB_SAVEDEFAULT=true
GRUB_DISABLE_SUBMENU=y
. . .

More info: https://www.digitalocean.com/community/tutorials/how-to-update-a-digitalocean-server-s-kernel#booting-to-a-non-default-kernel-on-grub-2-distributions Ubuntu 12.04, 14.04 and 16.04 uses Grub 2

15.2.5. Shutdown Ubuntu

shutdown -h now

15.3. Environment and Shell Variables

  • Child processes inherit the environmental variables of the parent process
  • printenv or env
  • printenv HOME or echo $HOME
  • VARNAME="value"
  • export
    Set an env. variable for the current shell and all of its child processes
    export TERM=xterm-256color
    See a list of env. variables that are manually set
    export -p
    Set an env. variable using a shell variable
    export TEST_VAR
    Concatenate
    export PATH=/a/new/path:$PATH"
  • /etc/environment
    • Set permanently and system wide (all users, all processes) add set variable in /etc/environment sudo nano /etc/environment, then logout from current user and login again
  • env
    • Modify the environment that programs run in
      • env VAR1="blahblah" command_to_run command_options
  • set
  • (set -o posix; set)
  • Shell and env. variables should in uppercase while local variables should be in lowercase
  • TEST_VAR='Hello World!'
  • printenv | grep TEST_VAR
  • export -n TEST_VAR
  • unset TEST_VAR
  • http://en.tldp.org/LDP/abs/html/internalvariables.html
    MACHTYPE
    machine type. Used to distinguish OS platforms e.g. apple or linux?
    HOSTNAME
    system/computer name
    HOME
    home directory
    (no term)
    SHELL
    (no term)
    BASH_VERSION
    SECONDS
    number of seconds the Bash session has run. Good for bash script running time
    $0
    the script file name in which $0 is called
    PWD
    current directory
    OLDPWD ~-
    previous dir cd - echo ~!
    _
    last argument of previous command executed. echo $_
    ls -al >/dev/null && echo $_
    last argument is -al
    du >/dev/null
    last argument is du
    (no term)

    History 

    export HISTSIZE=10000  # default 500
export HISTFILESIZE=1000000
export HISTTIMEFORMAT='%b %d %I:%M %p ' # using strftime format
export HISTCONTROL=ignoreboth # ignoredups:ignorespace
# ignorespace means if the command starts with a space, then it won't be saved in history
export HISTIGNORE="history:pwd:exit:df:ls:ll"
    delete a line (123), doesn't work for zsh
    history -d 123
    (no term)
    For Zsh
    Method 1
    nano ~/zsh_history you will need to restart iterm terminal
    Method 2
    LC_ALL=C sed -i '' '/porn/d' $HISTFILE and source ~/.zshrc

15.4. Bash - Bourne-Agin Shell

15.4.1. Config

  • Login shell
    • A login shell is invoked when
      • bash is invoked as in interactive login shell /bin/bash -i --login
      • or as a non-interactive shell /bin/bash --login
    • a login shell is loaded when X11 or virtual terminal is loaded
      • e.g. Open Terminal on Mac using Bash
      • --noprofile option may be used when the shell is started to inhibit this behavior

    • Loading sequence

      • /etc/profile

      • Ubuntu

        • If $BASH is not /bin/sh then load /etc/bash.bashrc (interactive bash shells)
        • Load files in /etc/profile.d
        • and First found of the following
      • ~/.bash_profile
        • Ubuntu doesn't have this file
      • ~/.bash_login
        • Ubuntu doesn't have this file
      • ~/.profile

        • Ubuntu

          • If running bash, include ~/.bashrc (interactive non-login shells)
          • ~/.bash_aliases
            Add user's private bin to $PATH
            ~/bin ~/.local/bin
      • ~/.login for Mac Bash
    • Logout
      • ~/.bash_logout
  • Subshell
    • When an interactive shell that is not a login shell is started
    • /etc/bash.bashrc and ~/.bashrc are run
      • --norc option may be used to inhibit
      • --rcfile option will force bash to run a certain file rather than the default 2 files
  • Graphical shells, in Ubuntu, will read /etc/profile and ~/.profile but not for all Linux
  • In Ubuntu, .bashrc is loaded when
    • a login and interactive shell
    • Subshell

  • Since ~/.bashrc may not be included in all Linux distros and all login/subshell instances, may need to manually include it e.g. inside ~/.bash_profile 

    if [ -f ~/.bashrc ]; then
    . ~/.bashrc
fi

15.4.2. Prompt, Bash Prompt

  • Variables
    \$
    shows $ if current user is normal or # for root user
    \u
    username
    \w
    current working dir with tilde

Color

  • \[\033[COLOR]m\] e.g. PS1="\[\033[31m\]\u@\h:\w$ "
  • 31,red 32,green 30,black 37,white 33,yellow 35,purple
  • 00m means to reset both text style and color
  • More colors

Text Style

  • \[\033[ATTRIBUTE; COLORm\] e.g. PS1="\[\033[4;31m\]\u@\h:\w$ "
  • PS1="\[\033[01;04;31m\]\u@\h:\w$ "
  • normal
  • bold for Ubuntu
  • dim text
  • underline
  • blinking
  • reverses text and background colors
  • hide text

Run Command

  • PS1="\u@\h on `id -gn` \w\$ " where id -gn is a command
15.4.2.1. bash-git-prompt

https://github.com/magicmonty/bash-git-prompt 

# install on Ubuntu
git clone https://github.com/magicmonty/bash-git-prompt.git ~/.bash-git-prompt --depth=1

# add to ~/.bashrc
GIT_PROMPT_ONLY_IN_REPO=1
source ~/.bash-git-prompt/gitprompt.sh

source ~/.bashrc

# see available Ubuntu themes
git_prompt_list_themes | grep Ubuntu

# change theme in .bashrc
GIT_PROMPT_THEME=Solarized

# generate a custom theme filed after theme is set to Custom in .bashrc
# generate a custom theme based on the Single_line_Ubuntu theme
# ~/.git-prompt-colors.sh is created and later you will customize it
git_prompt_make_custom_theme Single_line_Ubuntu

# in .bashrc
GIT_PROMPT_THEME=Custom

# toggle gitprompt
git_prompt_toggle

15.4.3. Alias

15.4.4. Shortcuts

  • https://github.com/fliptheweb/bash-shortcuts-cheat-sheet
  • Process
    Exit command line when it's empty
    C-d
  • Screen
    clear screen
    C-l
    Stop all output of the currently running foreground command without terminating it
    C-s
    Resume output of the above
    C-q
  • Navigation
    Go to command start/end
    C-a C-e
    Go forward/backward one word
    M-f, Esc-f / M-b Esc-b
    Go forward/backward one character
    C-f C-b. As bash:tmux uses C-b, double C-b to achieve the same
    Move to the beginning and hit again to go back the original cursor position
    C-xx
  • Delete or cut (add to clipboard)
    Delete or cut from the beginning to current cursor
    C-u
    Delete from the current cursor to the end
    M-d
    Cut from the current cursor to the end
    C-k
    Cut the word before the cursor
    C-w
    Delete one character at cursor
    C-d
    Delete one character before cursor
    C-h
  • Paste
    yank
    C-y
  • Undo
    Undo last key press
    C-_
  • Swap
    Swap the current word with the previous word
    M-t
    Swap the last 2 characters order
    C-t
  • Capitalizing
    Capitalize from cursor to the end of the current word
    M-u
    Uncapitalize from cursor to the end of the current word
    M-l
    Capitalize the character at cursor, but cursor will move to the end of the current word
    M-c
  • History
    Go to previous command
    Up or C-p
    Go to next command
    Down or C-n
    Revert to a command you've pulled from history if it's edited
    M-r
    Recall the last command by searching

    C-r

    Run a command you found with C-r
    C-o
    (no term)
    Leave C-r without running a command
    Search forward
    C-r
    Trick
    add a comment and later search that comment
    • e.g. run ls -al #longcommenttoIDthiscommand
    • C-r and type #longcommenttoIDthiscommand
    Return last run command without running it
    echo !!
    Run last run command
    !!
    run 2nd last command
    !-2
    run command at a number
    !123
    Return last run command starting with cat and put this command to the last run command history
    !cat:p
    Use last run command's arguments
    mkdir /new/path then cd !$
    Modify last run command by replacing the first instance of nanp with nano
    nanp .ssh/config then ^nanp^nano
    Run the 455th command in history
    !455
    (no term)
    Run a command with several arguments that have minor difference using curly brackets {}
    • mv /path/to/file.{txt,xml}
    • mkdir myfolder{1,2,3}
    • cp /etc/rc.conf{,-old} eq. to cp /etc/rc.conf /etc/rc.conf-old
    • mv /etc/rc.conf{-old,} eq. to mv /etc/rc.conf-old /etc/rc.conf

15.4.5. Here Documents 

# EOF is not quoted means parameter expansion and command substitution and arithmetic expansion is possible
# \ must be used to quote the characters \, $ and `
mysql -u root -p mydb <<EOF
SELECT * FROM abc
WHERE name LIKE '%lili%'
EOF

# use <<-EOF or <<-'EOF' to strip tab characters in the actual input

# EOF is quoted, no expansion
mysql -u root -p mydb <<'EOF'
SELECT * FROM abc
WHERE name LIKE '%lili%'
EOF

15.4.6. Options

# List of all options and their on/off status
shopt

# See if an option is enabled or not
shopt extglob

# List of all enabled options
shopt -s

# List of all unset/disabled options
shopt -u

# Enable an disabled option
shopt -s histappend

# Disable all options
shopt -u

# Eanble a disabled option
shopt -u cmdhist

# Enable extglob for just the running command: prints all immediate child directories and files whose name is not foo
bash -c -O extglob 'echo !(foo)'

15.5. zsh

15.5.1. Oh My Zsh

  • Basics 

    # install script from https://github.com/ohmyzsh/ohmyzsh
sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

zsh --version

# .zshrc is created or modified

# manually upgrade zsh
upgrade_oh_my_zsh

# update changes
source ~/.zhrc

brew install zsh-autosuggestions
  • Inside ~/.zhrc
    • ZSH default folder: ~/.oh-my-zsh
      • This folder is set as ZSH folder
      • can be used to change the default folder: ~/path/to/custom
      • Sub directories
        1. lib
          • Oh My Zsh config files broken down by module e.g. history.zsh and source $ZSH_CUSTOM/lib/history.sh is run
          • Don't change these files
        2. plugins
        3. custom
          • Create any *.zsh directly in the folder e.g. ~/.oh-my-zsh/custom/alias.zsh
        4. themes
    • source $ZSH/oh-my-zsh.sh
      • Initialize Oh My Zsh
    • Followed by your own custom settings
  • https://www.freecodecamp.org/news/how-to-configure-your-macos-terminal-with-zsh-like-a-pro-c0ab3f3c1156/
15.5.1.1. Built-in plugins
copyfile
copyfile path/to/file to copy the file into clipboard
(no term)
macos
ofd
open the current directory in a Finder window
pfd
return the path of the frontmost Finder window
cdf
cd to the current Finder directory
(no term)

jsontools 

# pretty print
echo '{"b":2, "a":1}' | pp_json

# echo two separate json objects and pretty print both
echo '{"a": "b"}\n{"c": [1,2,3]}' | pp_ndjson
15.5.1.2. Command alias
d
eq. dirs -v list last visited directories
cd -1
go to directory number 1
cd -2
-
cd to last visited directory
cd or ~
cd ~
...
cd ../..
....
cd ../../..
.....
cd ../../../..
/
cd /
md
mkdir -p
mkcd
create a new directory and change to it
take URL or take path/to/directory
like mkcd and it handles remote URLs. Download the file and extract it and go to the newly extracted/downloaded/cloned directory
./ + tab + tab
then use arrow keys to view and select a file or directory under the current directory

15.5.2. iTerm2

15.5.2.1. Install
# install or upgrade CLI developer tools for Xcode
xcode-select --install

# To get the current Xcode version (when only CommandLineTools is installed)
pkgutil --pkg-info=com.apple.pkg.CLTools_Executables | grep version

# if an error, run this to reset `xcode-select`
# xcode-select -r

# install homebrew

brew install --cask iterm2

# download Color Presets
cd Downloads
curl -O https://raw.githubusercontent.com/MartinSeeler/iterm2-material-design/master/material-design-colors.itermcolors

# Go to iTerm2 > Preferences > Profiles > Colors Tab
# Click Color Presets… at the bottom right
# Click Import…
# Select the material-design-colors.itermcolors file
# Select the material-design-colors from Load Presets
15.5.2.2. Settings ⌘ + ,
  • Profiles
    • General
      Working Directory
      Reuse previous session's directory
    • Advanced
      • Visual bell flashes the whole screen, not just a bell icon
15.5.2.3. Shortcuts
  • Tab
    new tab
    Cmd-t
    close current tab
    Cmd-w
    Switch to a tab left or right
    Cmd-S-[ Cmd-S-]
    Switch to a tab
    Cmd-1, Cmd-2
    Edit Session
    ⌘ i
    Tab title
    Change tab name
  • Pane
    new vertical pane
    Cmd-d
    new horizontal pane
    Cmd-S-d
    Close current pane
    C-d
    Switch to a pane left or right
    Cmd-[ Cmd-]
  • ⌘ ;
  • Preference > Profiles > Keys > Key Mappings
    • Use preset Natural Text Editing
      • Add
        • M-b as Esc-b
        • M-f as Esc-f
15.5.2.4. Recent Directories ⌘ ⌥ /
15.5.2.5. Terminal Replay :: ⌥ ⌘ b and then use arrow keys and hold ⇧ plus arrows to replay faster
15.5.2.6. Show execution time ⇧ ⌘ e

15.5.3. .zshrc

  • get some ideas from `cat /etc/zshrc`
15.5.3.1. Customize Prompt
man zshmisc

# Scroll to section SIMPLE PROMPT ESCAPES to get all variables / Escapes which can be used in Prompt

nano ~/.zshrc

PROMPT="%(?:%{$fg_bold[green]%}➜ :%{$fg_bold[red]%}➜ )"
PROMPT+=' %{$fg[cyan]%}%~%{$reset_color%} $(git_prompt_info)'

source ~/.zshrc

15.5.4. .zsh_history

  • Delete a command from history

15.5.5. Options 

# All enabled options
setopt

# All disabled options
unsetopt

15.6. User Management, Superuser, sudo

15.6.1. Create and delete a user, Add user to sudo group

  • sudo adduser corey
  • sudo deluser corey
  • Use useradd for non-interactive (Docker)
    • Example
      • useradd --user-group --create-home newusername

    • Options

      --usergroup, -U
      Create a group with the same name as the user
      --create-home, -m
      Create home directory if does not exist
      --shell, -s
      Assign a default shell for this user. /bin/false returns false then exit (logout)
      • useradd --user-group --create-home --shell /bin/false newusername
        -o -u 1000
        change UID to 1000, -o is to allow duplicate UID
15.6.1.1. Add user to the sudo group
  • See 15.6.2 to fine tune permissions

As root user, run this to add the user to the sudo group 

gpasswd -a username sudo
# or
usermod -aG sudo username
# -aG keeps the existing groups the user belongs to and add the user to another group

# list groups a user is in
groups username

# list groups for the current user
id -nG
# or
groups

sudo usermod -aG agroupname ausername

15.6.2. Sudoer file

  • Run sudo visudo to see the sudoers file to check:
    • permissions given to each group
    • check sudo environment variable setting
  • Instead of directly editing using sudo visudo (don't directly edit file /etc/sudoers), add files to /etc/sudoers.d
  • Examples
    root ALL=(ALL:ALL) ALL
    User root can run sudo as all hosts (1st ALL), all users, all groups and apply to all commands
    %sudo ALL=(ALL:ALL) ALL
    Start with % means is a group
    lili2 ALL=(ALL:ALL) ALL
    lili2 can run any command or login as any user
    ubuntu ALL=(ALL) NOPASSWD:ALL
    ubuntu can run any command without using password on any host or login as any user
  • groups username
  • More info

15.6.3. Change a user's password

  • Login as root and run passwd to change root password
  • change another user's password

15.6.4. List all users cut -d: -f1 /etc/passwd or cat /etc/passwd linux:/etc/passwd

  • Login username
  • Password (always x)
  • UID (<500 are system accounts, > 500 are users)
  • Group ID (GID)
  • Comment field
  • Location of HOME directory
  • Default shell for the user

Refer to bash:awk awk -F':' 'BEGIN{OFS="\t"; print "Login Username","Password","UID", "Group ID (GID)", "Comment", "Home Dir", "Default shell"} {print $1,$2,$3,$4,$5,$6,$7}' /etc/passwd | column -t -s $'\t'

15.6.5. Sudo

  • Options
    -S, --stdin
    linux:sudo:password
    -H
    use the target user's home directory
    -n, --non-interactive
    avoid prompting the user for input of any kind. If a password is required, it will stdout an error message and exit
15.6.5.1. Switch to another user
  • Switch to superuser/root
    Debian
    sudo su
    RedHat
    sudo -
  • Login to another user, Run command as another user
    • Login (need to type password)
      • /bin/su - username or su liliadmin
      • exit to go back to original user
    • Run as a user (need to type password)
      • sudo -u lynda whoami
      • /bin/su - username -c 'whoami'

    • As root user, run as another user without typing password

      • sudo -H -u username bash -c 'whoami'
15.6.5.2. Sudo run multiple commands

sudo bash -c 'apt-get update;apt-get install'

15.6.5.3. Pass password as a file to sudo

sudo -S your_script -var1 </path/to/file.txt

15.6.5.4. Require no password when sudo is run as another user
  • User abc has sudo privilege. When abc is signed in and run a script/command that requires sudo, system will ask for password
    • e.g. sudo ./path/to/script.sh This bash file can contain make start where recipes may not have sudo in front
    • bypass so password input is not required

  • Instead of changing /etc/sudoers file directly, create/edit a file /etc/sudoers.d/mybypass 

    # See if there are some sample
sudo cat /etc/sudoers
sudo ls -al /etc/sudoers.d/

# found one sample
sudo cat /etc/sudoers.d/mybypass

# have to use visudo to edit /etc/sudoers and /etc/sudoers.d/anyfile files.
# -f is to specify a file, otherwise is the default /etc/sudoers file
sudo visudo -f /etc/sudoers.d/mybypass

# Add this line
abc ALL = (root) NOPASSWD: /path/to/script.sh
# allow user abc on ALL hosts to run a specific command without entering password
# all other sudo commands will still require a password

# ~/etc/sudoers.d/mybypass file should have permission 0440 and owner:group is root:root

15.7. SSH, scp, sshpass

See

15.7.1. Install SSH Server

Ubuntu 

sudo apt install ssh

# should be started
service ssh status

sudo service ssh stop
sudo service ssh start

# disable ssh service
sudo systemctl disable ssh

sudo systemctl enable ssh

15.7.2. Create .ssh folder

cd ~
mkdir .ssh
chmod 700 ~/.ssh && chmod 600 ~/.ssh/*

15.7.3. Generate client public and private keys, transfer public key to remote

  • Generate a private key on local machine ssh-keygen -t rsa -b 2048
    -t rsa
    type of key
    -b 2048
    bits
    (no term)
    -f myfilename
    myfilename file
    private key
    myfilename.pub file
    public key
    (no term)
    -P existingPassPhrase
    -p
    request changing the passphrase of a private key file instead of creating a new private key
  • Under folder /Users/username/.ssh/ id_rsa.pub is the public key and id_rsa is the private key
  • cat ~/.ssh/id_rsa.pub
  • When using private key you might encounter Permissions 0777 for '~/.ssh/id_rsa_your_private_key' are too open

Use chmod 400 ~/.ssh/id_rsa_your_private_key to fix it chmod:400

  • Move public key to remote
    • Fast way
      Append the client public key to remote host's /.ssh/authorized_keys
      ssh-copy-id remoteusername@123.45.56.78
      Or specify a public key file
      ssh-copy-id -i ~/.ssh/id_rsa.pub remoteusername@123.45.56.78
  • Show fingerprint of specified public key. Private RSA1 keys are also supported
    • ssh-keygen -lf publickey
15.7.3.1. Manual way move pub key to remote
  • On the remote server, login as a normal user sudo su - username
  • Create ~/.ssh directory mkdir .ssh && chmod 700 .ssh
  • Create file and paste the public key nano .ssh/authorized_keys

Or concatenate it onto authorized_keys file manually 

# backup first
cp authorized_keys authorized_keys_Backup
cat id_rsa.pub >> authorized_keys

Change permission chmod 600 .ssh/authorized_keys chmod:600

Add a name to id_rsa.pub 

cat yourpubkey.pub
ssh-rsa longstring yourname@yourhost
15.7.3.2. Move private key to another computer
  • mkdir .ssh chmod 700 .ssh
  • cd .ssh && nano yourprivate_key
  • chmod 400 yourprivate_key
15.7.3.3. Remove passphrase for private key id_rsa
cd ~/.ssh
ssh-keygen -p -f id_rsa
# enter old passphrase and new

The following no longer works 

# Make a backup
cp ~/.ssh/id_rsa ~/.ssh/id_rsa.backup
rm ~/.ssh/id_rsa

# remove passphrase
openssl rsa -in ~/.ssh/id_rsa -out ~/.ssh/id_rsa_new

# enter old passphrase
cp ~/.ssh/id_rsa_new ~/.ssh/id_rsa

15.7.5. Config linux:ssh:config

Create ssh host alias and the private key to use in ~/.ssh/config file 

cd ~/.ssh && nano config
chmod 600 ~/.ssh/config

Host *
  ServerAliveInterval 240
  ServerAliveCountMax 2

Host scotch
    HostName scotch.io
    User nick

Host example2
    HostName example.com
    User root
    Password abc

Host example3
    HostName 64.233.160.0
    User userxyz123
    Port 56000
    ServerAliveInterval 240
    ServerAliveCountMax 2

Host example4-Switch-to-root
    HostName 64.233.160.0
    User nonRootUserThatIsSudoerCanSwitchToUser
    # This solution still doesn't provide the root user for SFTP through SSH.
    # SFTP still uses the non-root user
    # But the solution will change the root once logged in
    RemoteCommand sudo su -
    # optional. Required if remote requires TTY
    RequestTTY yes

Host amazon1
    HostName ec2.amazon.com
    User ec2-user
    IdentityFile /path/to/special/privatekey/amazon.pem
ServerAliveInterval 240
Sends a packet to the server every 240 seconds (4 minutes). If the client does not receive a response after 2 tries, it closes the connection.
  • In Putty, under Connection > Seconds between keepalives > 240
  • To kill the connection when it gets hang up, ~.
(no term)
Convert Putty private key .ppk to OpenSSH using puttykeygen
  • Load > Conversions (in top menu): Export OpenSSH key, save as id_rsa_yourweb without file extension

15.7.6. Remove a known_host

  • TS: SSH Client, Warning Remote Host Identification Has Changed!
  • ssh-keygen -f "/home/vagrant/.ssh/known_hosts" -R "[172.22.0.3]:2222"
  • ssh-keygen -f "/home/vagrant/.ssh/known_hosts" -R "[172.22.0.3]:0" use 0 for all ports

15.7.7. Use private key from SSH server to connect

AWS uses .pem file as the private key on its SSH server. Convert this .pem file in PuttyGen and later uses it to connect to AWS Refer to aws:ssh

15.7.8. Restart SSH

  • Ubuntu
    • See 15.7.9.1 (22.10)
    • sudo systemctl reload sshd (16.04)
    • sudo systemctl restart sshd (16.04)
    • sudo restart ssh (14.04)
  • service ssh restart

15.7.9. sshd_config SSH Daemon Config File

15.7.9.1. Socket-activated
  • Since Ubuntu 22.10, SSHd uses socket-based activation
  • Some settings on sshd_config is not used any more
  1. Add port
    • Add port 443 on top of the default port 22
    mkdir -p /etc/systemd/system/ssh.socket.d
cat >/etc/systemd/system/ssh.socket.d/listen.conf <<EOF
[Socket]
ListenStream=
ListenStream=443
EOF

sudo systemctl daemon-reload
sudo systemctl restart ssh
# Should list `Server listening on :: port 22` and 443
systemctl status ssh
15.7.9.2. Backup
sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.factory-defaults
sudo chmod a-w /etc/ssh/sshd_config.factory-defaults

15.7.10. Disallow SSH as root user, SSH Daemon Config linux:ssh:disallow_root

  • Login as root then nano /etc/ssh/sshd_config
  • Change PermitRootLogin yes to no
  • Restart SSH
  • Before logout, open another terminal to see if it works
  • Default SSH port is 22

15.7.11. Disable Password Authentication

Force to use public key only instead of plain password only
sudo nano /etc/ssh/sshd_config
(no term)
Replace #PasswordAuthentication yes with PasswordAuthentication no
(no term)
linux:ssh:restart
(no term)

These are supposed to be default. If not, change to as below 

PubkeyAuthentication yes
ChallengeResponseAuthentication no

15.7.12. Login without setting public keys bash:sshpass

https://www.cyberciti.biz/faq/noninteractive-shell-script-ssh-password-provider/ 

apt-get install sshpass

sshpass -p 't@uyM59bQ' ssh -o StrictHostKeyChecking=no username@server.example.com

# StrictHostKeyChecking is for shell script to ignore prompt

# Use a file for password, this is the most reliable way when it has special characters for rsync
echo 'myPassword' > myfile
chmod 0400 myfile
sshpass -f myfile ssh vivek@server42.cyberciti.biz

# scp
sudo sshpass -p 'abc' scp -o StrictHostKeyChecking=no -r /local/path/archive/ user@1.2.3.4:/remote/path/dtp-tmp/

# use with rsync
rsync --rsh="sshpass -p myPassword ssh -l username" server.example.com:/var/www/html/ /backup/

# ssh -l means to specify a username to login

# use environment variable SSHPASS to store password and it has to SSHPASS
SSHPASS='yourPasswordHere'
rsync --rsh="sshpass -e ssh -l username" server.example.com:/var/www/html/ /backup/

# gpg encrypted file
echo 'mySshPasswordHere' > .sshpassword
gpg -c .sshpassword
rm .sshpassword
gpg -d -q .sshpassword.gpg > fifo; sshpass -f fifo ssh vivek@server1.cyberciti.biz

15.7.13. Random password

# base64 will likely to have = at the end, remove those to have length 16
openssl rand -base64 17

# in the middle there may be + or / characters, try the following to remove those. Ensure the first number 20 is larger than the last number which is the final length
openssl rand -base64 20 | tr -d "=+/" | cut -c1-16

# hex has only number and lowercased letters
openssl rand -hex 16

15.7.14. Run command

# string does not need to be escaped
dbpw='$@$$$'
filedate=$(date +%FT%H%M%S)
ssh user1@server1 command1
ssh user1@server1 'command2'
ssh user1@server1 'command1 | command2'

ssh usehostinssh-config 'long command'

# escape single quote inside command
# wrong
ssh host1 'mysqldump -u user --password='$(dbpw)' dbname > ~/db.sql'
# correct:
ssh host1 'mysqldump -u user --password='"'"'$(dbpw)'"'"' dbname > ~/db.sql'

# '"'"' ::
# ' end the first quotation
# " start second quotation
# ' quoted character
# " end second quotation using double-quotes
# ' start third quotation

# result
# mysqldump -u user --password='abc' dbname > ~/db.sql

# You might not want to force to use single quote
ssh host1 "mysqldump -u user --password='$dbpw' dbname > ~/db-$filedate.sql"

# Save dump to local
ssh host1 "mysqldump -u user --password='$dbpw' dbname" > ~/db-live-"$filedate".sql

15.7.15. SSH tunneling

  • TCP Port forwarding is also called tunneling
  • Say your IMAP port 143 to gmail is blocked by Wi-fi firewall
  • Default ssh port is 22. Default https port is 443

    • For Github and BitBucket, you can directly change to use port 443 without doing anything 

      Host github.com
  Hostname ssh.github.com
  Port 443

Host bitbucket.org
  Hostname altssh.bitbucket.org
  Port 443
    • If you have control of the target server's SSH server, simply just add Port 443 to /etc/ssh/sshd_config and use -p 443 locally
      • See 15.7.9.1 for Ubuntu 22.10+ to add port 443
    • If you don't have control of the target server's SSH server
      Assume you have a Host entry for intermediate host
      myserver using port 443
      (no term)
      In your intermediate host my.server.com, add Port 443 as a second SSH port and set AllowTcpForwarding yes in /etc/ssh/sshd_config, 15.7.8
      AllowTcpForwarding
      yes (default, means, allow all forwarding local / remote to allow local/remote forwarding)
      (no term)
      Optional
      AllowStreamLocalForwarding
      yes (default is not defined)

    • On your local set another Host entry in ~/.ssh/config 

      Host targetserver-localhost
    Hostname 127.0.0.1
    Port 10143
    User myyser
    IdentityFile ~/.ssh/id_rsa_targetserver
    RemoteCommand sudo su -
    RequestTTY yes

Host targetserver
    Hostname target.server.ip.0
    User myuser
    IdentityFile ~/.ssh/id_rsa_targetserver
    RemoteCommand sudo su -
    RequestTTY yes
      And initiate the local port forwarding on local
      ssh -L 10143:target.server.ip.0:22 myserver
      • Any traffic to localhost:10143 is forwarded to target.server.ip.0 at port 22 through myserver at port 443
      (no term)
      Now you can ssh targetserver-localhost on local
15.7.15.1. Local Port Forwarding
  • We are going to forward local client port 10143 to an intermediate host my.server.com at port 443, which forwards to mail.google.com at 143
    • In the intermediate host with ssh server installed, add the line Port 443 to /etc/ssh/sshd_config, linux:ssh:restart, sudo ufw allow 443/tcp
    • Check if it connects from local ssh -p 443 my.server.com

    • Then open, listen and forward local client port 10143 to destination imap.google.com at port 143 using intermediate host at 443 

      ssh -L localhost:10143:mail.google.com:143 -p 443 user@my.server.com
    • Change local email client setting to use localhost at port 10143
  • See network:port on port range you should use
    See if port 10143 is being used on localhost
    lsof -i :10143

  • Now SSH to localhost:10022 will be forwarded to target_server_user@target.server.com:22 through SSH server user@my.server.com:443 

    ssh -p 10022 target_server_user@localhost
scp -P 10022 localhost:/target/server/path/ /local/path/
rsync -av -e "ssh -p 10022" /target/server/path localhost:/local/path/
  • Local forwarding. specifies that the given port on the local (client) host is to be forwarded to the given host and port on the remote side. Allocate a socket to listen to port on the local side, optionally bound to the specified bind_address. Whenever a connection is made to this port, the connection is forwarded over the secure channel, and a connection is made to host port hostport from the remote machine.
    Summary
    [localhost | bind_address]:port > -p 443 user@my.server.com (SSH) > host:hostport (access from my.server.com)
    (no term)

    Multiple redirections can be set up in the same tunnel 

    # add -f to go into background and -N to not launch a shell
ssh -L localhost:10143:mail.google.com:143 localhost:10022:target.server.com:22 -p 443 user@my.server.com
  • -D [bind_addres:]port
    • Similar to -L but the destionation host:hostport is not specified. It's called dynamic application-level port forwarding.
    • e.g. ssh -CND 10022 -p 443 user@my.server.com
    • Now a SOCKS proxy server is set up. SOCKS is a protocol
      • 127.0.0.1:10022
      • localhost:10022
    • Set up application e.g. Firefox, Chrome to use the SOCKS 5 tunnel
      • Firefox

      • Chrome 

        "C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" ^
  --user-data-dir="%USERPROFILE%\my-proxy-profile" ^
  --proxy-server="socks5://127.0.0.1:10022" 

# or chrome.exe
/usr/bin/google-chrome \
  --user-data-dir="$HOME/my-proxy-profile" \
  --proxy-server="socks5://127.0.0.1:10022" 

"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
  --user-data-dir="$HOME/my-proxy-profile" \
  --proxy-server="socks5://127.0.0.1:10022" 

# --user-data-dir :: user profile will be created automatically if not exists

# The hostname for destination URLs will be resolved by the proxy server.
# ftp:// URLs though a SOCKS proxy is not implemented

# To prevent Chrome from sending any DNS requests over the network (e.g. DNS prefetch uses direct DNS resolve even though proxy is used)
# --host-resolver-rules="MAP * ~NOTFOUND , EXCLUDE myproxy"

# Verify proxy setting
# chrome://net-internals/#proxy

# Verify DNS resolving
chrome://net-internals/#dns

# Verify proxy logic for individual requests
# chrome://net-internals/#events

# Use HTTP proxy "foopy:80" for http URLs and HTTP proxy "foopy2:80" for ftp URLs
# Without scheme and a single proxy :: use that proxy for all URLs
chrome.exe --proxy-server="http=foopy:80;ftp=foopy2"

# --proxy-bypass-list=(<trailing_domain>|<ip-address>)[:<port>][;...]
# * can be used

# --proxy-server="foopy:8080" --proxy-bypass-list="*.google.com;*foo.com;127.0.0.1:8080"

      • Git 

        git config --global http.proxy 'socks5://127.0.0.1:10022'
git config --global https.proxy 'socks5://127.0.0.1:10022'
    • For applications that don't support using SOCKS

      • Use torify or usewithtor (installed along with tor) as shell wrapper to re-direct command line traffic using SOCKS https://neverendingsecurity.wordpress.com/2015/03/14/proxify-applications-with-torsocks-tsocks-and-proxychains-on-linux/ 

        sudo apt-get install tor

# re-direct for `telnet google.com 80`
torify telnet google.com 80

# or

usewithtor telnet google.com 80

# On top of wrapping command line, some applications/software are supported

userwithtor pidgin

      • tsocks 

        sudo apt-get install tsocks

nano /etc/tsocks.conf

# server = 127.0.0.1
# server_port = 100220

# Run applications using tsocks. Run locally (applications, command line e.g. ~ftp~ ~ssh~, you don't need to open different SSH tunnels/ports!):
tsocks skype tsocks kopete tsocks kmail tsocks ftp someserver.com tsocks ssh user@target.server.com

# Notes
# - FTP has to be in passive mode
# - Some applications might not support tsocks
      • sshuttle
  • Other options
    -N
    Don't run any command on user@my.server.com after the connection is first established
    -f

    fork the process to the background 

    # To terminate
ps aux | grep ssh

# Find the PID by searching the command

kill $thePID
    -C
    compress the data before sending
    -q
    quiet mode
15.7.15.2. Remote Port Forwarding
  • -R [bind_address:]port:host:hostport user@my.server.com
    • Remote port forwarding (on local, initiate remote port fording for my.server.com). Specifies that the given port on the remote (server) host is to be forwarded to the given host and port on the local side. This works by allocating a socket to listen to port on the remote side, and whenever a connection is made to this port, the connection is forwarded over the secure channel, and a connection is made to host port hostport from the local machine.
    • [bind_address | empty means any IP] (access from internet) > port at my.server.com > -p 443 user@my.server.com (SSH) > host:hostport (usually host is localhost)
    • host:hostport is usually localhost:8888
    • e.g. traffic from the internet access my.server.com:9000 will be forwarded to your localhost:8001. Run this locally, and the SSH server my.server.com will open a port 9000 which forwards request to your localhost:8001
      • ssh -R 9000:localhost:8001 -p 443 user@my.server.com
      • linux:ssh:restart">GatewayPorts yes
        GatewayPorts yes
        my.server.com allows anyone to connect to port 9000. SSH server my.server.com will open a port 9000 for anyone
        Anyone
        If the server is on the public internet, anyone on the Internet can connect to port 9000
        GatewayPorts clientspecified
        e.g. only connections from the IP address 1.2.3.4 to port 9000 is allowed
        • ssh -R 1.2.3.4:9000:localhost:8001 -p 443 user@my.server.com
  • allows remote hosts to connect to local forwarded ports.
  • don't run command. Just forwarding ports.
15.7.15.3. SOCKS on Windows

Windows Internet Options can set SOCKS5 as proxy. However, DNS will be resolved locally at client. While Chrome and FireFox support resolving DNS on the proxy DNS.

On Windows, use the portable SocksCap64, run it as admin, setup SOCKS5 proxy, and add individual Windows application as executable, select the app and click Run!

15.7.16. TS - WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!

  • The unique fingerprint of the destination server does not match what was stored in the client SSH known_hosts. Common reasons on destination server
    • root password is changed
    • It is rebuilt
    • Same IP address for multiple remote systems
  • delete the entries corresponding to the destination server's IP in known_hosts
    Linux and MacOS
    .ssh/known_hosts

15.8. fail2ban linux:fail2ban

Ban access to services (e.g. SSH) for max tries and ban time based on service's log.

Check if it's installed ll /etc/fail2ban

Install sudo apt-get update sudo apt-get install fail2ban Create jail.local based on jail.conf by commenting out every line 

awk '{ printf "# "; print; }' /etc/fail2ban/jail.conf | sudo tee /etc/fail2ban/jail.local

Modify jail.local sudo nano /etc/fail2ban/jail.local

By default, fail2ban is enabled for sshd service only and disabled for every service (allow every access) 

[DEFAULT]
ignoreip = 127.0.0.1/8

# 600 seconds = 10 minutes
bantime = 600
maxretry = 3

# 3 tries within 10 minutes
findtime = 600

# For a service, fail2ban can search if a log file has certain text and mark it as a authentication failure
[nginx-http-auth]
enabled = true

# regex pattern is defined in /etc/fail2ban/filter.d/ngix-http-auth.conf
# You usually don't modify these .conf files
Restart the service
sudo service fail2ban restart
(no term)
See which service has fail2ban enabled :; sudo fail2ban-client status
See detail about a service that has fail2ban enabled
sudo fail2ban-client status sshd
Unban an IP for service
sudo fail2ban-client set apache unbanip 111.111.111.111
(no term)
More info
(no term)
Apache and fail2ban

15.9. System

Resource Limits
sudoedit /etc/security/limits.conf
(no term)

System Logs

/var/log/syslog general system log
/var/log/auth.log system auth logs
/var/log/mail.log system mail logs
/var/log/messages general log messages
/var/log/dmesg kernel ring buffer msg, e.g. system bootup
   
Search a word `warthog` in .gz files
zgrep -i warthog /var/log/*.gz
(no term)
uptime
who
who are using the system right now

15.10. Filesystem

15.10.1. /dev /mnt /media

/dev/sd*1
sd for SATA or SCSI drive, numeric 1 for partition, * for a, b, etc.
/dev/hd*1
hd for IDE drive, * for a, b, etc.
(no term)
/media/win1
  • virtual directory where files are accessed file here.
  • Equivalent to d:\. One or more partitions can be assigned to one media
sudo mkdir /media/win2
create a virtual directory is easy
sudo mkdir /mnt/my_mount
also creates a virtual directory
sudo mount -t ext4 /dev/sda1 /mnt/my_mount
Mount a partitioned device to a virtual directory

sudo gedit /etc/fstab , /dev/hdb1 media/win1 vfat users,rw,owner,umask=000 0 0 , then reload sudo mount -a

  • Permanently mount during bootime

15.10.2. bindfs linux:bindfs

  • Tested on Ubuntu Xenial
  • Create a user devone and mount /home/devone/websites/application1 to /var/www/application1 which is owned by www-data
  • devone can create/edit files while the destination folder maintain the same owner www-data and permissions
apt-get update
apt-get -y install bindfs

sudo adduser devone
mkdir -p /home/devone/websites/application1
chown -Rf devone:devone /home/devone/websites
chmod -Rf 770 /home/devone/websites

nano /etc/fstab
bindfs#/var/www/application1 /home/devone/websites/application1 fuse force-user=devone,force-group=devone,create-for-user=www-data,create-for-group=www-data,create-with-perms=0644,chgrp-ignore,chown-ignore,chmod-ignore 0 0  

mount /home/devone/websites/application1
# if your system yells about force-user or force-group not being defined:
# replace force-user by owner
# replace force-group by group

# Test
su - devone
cd ~/websites/application1
touch hello.txt
ls -al hello.txt
# -rwxrwx--- 1 devone devone 0 sept. 10 17:15 hello.txt

exit
cd /var/www/application1
ls -al hello.txt
# -rwxrwx--- 1 www-data www-data 0 sept. 10 17:15 helloworld.txt

Make sure www-data user owns /var/www can prevent devone directly access the folder 

chown www-data:www-data /var/www
chmod 770 /var/www

If /etc/fstab is modified and you want to remount to reflect the changes. Unmount the existing first 

umount /home/devone/websites/application1
# if that fails, try to navigate out of the /home/devone/websites/application1 for all users
# or use -l for lazy and -f for force
# umount -l /home/devone/websites/application1
mount /home/devone/websites/application1

# or mount -a to mount all entries in /etc/fstabs

http://blog.netgusto.com/solving-web-file-permissions-problem-once-and-for-all/

15.10.3. /etc system configuration files

15.10.4. Disk space

df -h Disk free space on OS
  h for human-readable
du -sh Current folder size. -s shows total only
du -sh ./* List sizes of 1st level child folders and files
du -sh ./wp-content/* List sizes of 1st level child folders of one folder
   
Disk usage in current folder including all sub folders but excluding one sub folder
du --exclude=./wp-content/uploads --exclude=./wp-content/error_log -sh
Exclude a file?
du --exclude-from 'path/to/files.txt'
(no term)

--max-depth 

# Only get down to 1 level. When max-depth > 0, -s cannot be used
du --max-depth=1 -h

# sort by size in MB
du --max-depth=1 -h -m | sort -h

# Do not count subdirectories size
du --max-depth=1 -Sh
(no term)

Search which directory has taken up space, Highlight lines that have M or G in front for file size 

du -cha --max-depth=1 / | sort -h | grep -E "^[0-9\.]*[MG]"

# say /var has taken up space
du -cha --max-depth=1 /var | sort -h | grep -E "^[0-9\.]*[MG]"
(no term)
du options
-c, –total
have a grand total line at the bottom
-a, –all
write counts for all files, not just directories
-m
same as –block-size=1M
-B, –block-size=SIZE
SIZE can be K, M, G, T, P, E, Z, Y
-s
show total only
15.10.4.1. Large file deleted but still no space
Delete file reserved by process
sudo lsof / | grep deleted
(no term)
Either restart the service that is using the file or kill the process
check inodes usage. Delete old files to release inodes
sudo df -i /
15.10.4.2. /tmp is mounted as overflow (often sized at 1MB)

It's likely due to /tmp was not specified as its own partition and the root filesystem filled up and /tmp was remounted as a fallback After space has been cleared, just unmount the fallback and it should remount at tis original point. 

sudo umount overflow
# or
sudo umount /tmp
15.10.4.3. Swap
free
# if swap is not enabled, it shows
Swap: 0 0 0

# or another to see if swap is on, is to list
sudo swapon --show
  1. Swap partition is recommended
  2. Swap file

    https://www.digitalocean.com/community/tutorials/how-to-configure-virtual-memory-swap-file-on-a-vps

    linux:dd Swap file on Linux is a disk image e.g. /var/swap.img. Size is better 1 or 2x available system Ram. Run as root 

    cd /var
touch swap.img
chmod 600 swap.img

# fill the file with zeroes for 1gb or 1024mb
dd if=/dev/zero of=/var/swap.img bs=1024k count=1000

# or use this line to change file size
sudo fallocate -l 1G /var/swap.img

# initialize the swap filesystem
mkswap /var/swap.img

# enable swap file for the current boot
swapon /var/swap.img

# swapoff /var/swap.img

# Make it ready at boot, add a line to /etc/fstab, be careful!! Don't overwrite, just add a new line
echo "/var/swap.img none swap sw 0 0" >> /etc/fstab

# in case you run sudo
# echo "/var/swap.img none swap sw 0 0" | sudo tee -a /etc/fstab > /dev/null

# set how likely Linux virtual memory manager to use swap. 100 means it uses swap as much as possible and leave memory free.
# list all options
sysctl -a

# just VM options
sysctl -a | grep vm.

# or this way to see swappiness value
cat /proc/sys/vm/swappiness

# default seems to be 60 (you can also modify /etc/sysctl.conf file)
sysctl -w vm.swappiness=30

    /etc/fstab format

    var/swap.img :: File system specifier :: For disk-based file systems, either a device file name (/dev/sda1), a file system label specification (LABEL=), or a devlabel-managed symbolic link (/dev/homedisk) none :: Mount point :: Except for swap partitions, this field specifies the mount point to be used when the file system is mounted (/boot) swap :: File system type :: The type of file system present on the specified device (note that auto may be specified to select automatic detection of the file system to be mounted, which is handy for removable media units such as diskette drives) sw :: Mount options :: A comma-separated list of options that can be used to control mount's behavior (noauto,owner,kudzu) 0 :: Dump frequency :: If the dump backup utility is used, the number in this field controls dump's handling of the specified file system 0 :: File system check order :: Controls the order in which the file system checker fsck checks the integrity of the file systems

15.10.5. mkdir 

mkdir -m 777 my_dir_with_full_permissions
mkdir -pm 777 my_dir_with_full_permissions/{subfolder1,subfolder2}

15.10.6. File Permission

15.10.6.1. Return list of files in Octal 
# full table of info
stat filename

# just the octal file permissions
stat -c '%a' path/to/file

for f in $(ls -a); do stat -c "%a %n" $f; done;

# for a file or directory
stat -c "%a %n" /var/www/html
15.10.6.2. Change permission

  • Octal mode

    Permissions Binary Octal Description
    --- 000 0 No permissions
    –x 001 1 Execute only
    -w- 10 2 Write only
    -wx 11 3 Write and execute
    r-- 100 4 Read-only
    r-x 101 5 Read and execute
    rw- 110 6 Read and write
    rwx 111 7 full
           
    • Convert to Octal Mode
    • Convert to Permissions Bit

    • The first digit of the 4 digits in Octal mode

      Binary Octal Description
      000 0 setuid, setgid, sticky bits are cleared
      001 1 sticky bit is set
      10 2 setgid bit is set
      11 3 setgid and sticky bits are set
      100 4 setuid bit is set
      101 5 setuid and sticky bits are set
      110 6 setuid and setgid bits are set
      111 7 setuid, setgid, sticky bits are set
           
      SUID or setuid
      If set, when the file will be executed by a user, the process will have the same rights as the owner of the file being executed
    • If set, then replaces "x" in the owner permissions to "s", if owner has execute permissions, or to "S" otherwise. Examples:
      • -rws------ both owner execute and SUID are set
      • -r-S------ SUID is set, but owner execute is not set
    • Same as above, but inherits rights of the group of the owner of the file. For directories it also may mean that when a new file is created in the directory it will inherit the group of the directory (and not of the user who created the file)
      • If set, then replaces "x" in the group permissions to "s", if group has execute permissions, or to "S" otherwise. Examples:
        • -rwxrws--- both group execute and SGID are set
        • -rwxr-S--- SGID is set, but group execute is not set
      • It was used to trigger process to "stick" in memory after it is finished, now this usage is obsolete. Currently its use is system dependant and it is mostly used to suppress deletion of the files that belong to other users in the folder where you have "write" access to
      • If set, then replaces "x" in the others permissions to "t", if others have execute permissions, or to "T" otherwise. Examples:
        • -rwxrwxrwt both others execute and sticky bit are set
        • -rwxrwxr-T sticky bit is set, but others execute is not set
  • owner-group-public
    400
    r---–— e.g. ssh private keys in ~/.ssh/ chmod:400
    440
    read for owner and group, but none for public chmod:440
    555
    read and execute (can't modify) for all chmod:555
    700
    rwx-–— full for owner chmod:700
    755
    rwxr-xr-x (e.g. wordpress directory) Default directory permission and default executable file permission
    770
    rwxrwx— chmod:770
    775
    rwxrwxr-x chmod:775
    777
    -rwxrwxrwx Directory's full permission chmod:777
    600
    -rw-–— only owner can read and write file chmod:600
    644
    -rw-r–r– (e.g. wordpress .php files)
    660
    rw-rw-— (e.g. wp-config.php) chmod:660
    666
    anyone can read and write. File full permission. chmod:666
    664
    rw-rw-r– chmod:664
  • By default, umask is 0022
    A file's full permission is 666. So new file has this default permission
    666-022 = 644
    A directory's full permission is 777. So new directory has default permission
    777-022 = 755
  • umask
  • umask 0022

  • Alpha notation 

    # Remove write permission for all
chmod a-w filename
chmod ugo=rwx filename
chmod u=rwx,g=rw,o=r filename
chmod ug+w filename
    u
    user that owns the file
    g
    group that owns the file
    o
    other (everyone else)
    a
    all (everybody)

  • -R 

    # In general, don't use -R but use find-chmod or find-xargs-chmod chain instead
# - -R will
#     - Execute `fchmodat` system call for each file and sub folder. Hence, changing the ctime (file status modification time) for all its files and sub folders.

# By using find-chmod or find-xargs-chmod, it finds the files and sub folders that need to be changed and then change the permissions

cd /path/to/targetFolder

# - `find -print0`
#     - `find` will separate each line of file path with a null character (a zero byte)
#     - `xargs` must then be called with `-0` or `--null`
# - `xargs`
#     - Default command is `echo`
#     - `xargs -0` eq. `xargs -0 echo` 
# - `xargs echo`
#     - Result :: echo 'foundFile1.txt' 'foundFile2.txt'
# - `xargs -I {} echo {}`
#     - `-I` :: Run the `echo` command for each record of the previous `find` command
#     - Result :: echo 'foundFile1.txt'; echo 'foundFile2.txt'
# This is the best solution 
find . -type f ! -perm 666 -print0 | xargs -0 -I {} chmod 666 {}
# chmod 666 foundFile1.txt
# chmod 666 foundFile2.txt
# ...

# eq.
# However, the list of files might be too long and error pops up
# -bash: /bin/chmod: Argument list too long.
find . -type f ! -perm 666 -print0 | xargs -0 chmod 666
# chmod 666 foundFile1.txt foundFile2.txt

# eq.
# Same caveat as the above
find . -type f ! -perm 666 -exec chmod 666 {} +

# eq.
# Same caveat as the above
chmod 666 $(find . -type f ! -perm 666)

# eq.
# `find .` can be `find /path/to/targetFolder`
  • See wp:check file permissions
15.10.6.3. Change owner:group for a folder 
# change a sub folder and all its files and directories
sudo chown -R user:group aFolder

# change current folder
sudo chown -R user:group .

List files or directories that are not a user or group. See wp:file permissions 

find . -type d -not -group test -o -type f -not -user test
15.10.6.4. Copy permissions and owners from one file to another

file2 will have the exact same owners and permissions as file1 

chown --reference=file1 file2
chmod --reference=file1 file2
15.10.6.5. Create several users for SFTP and final file permissions are www-data linux:permission:users

If more than one users are needed, try linux:bindfs

Otherwise, add user to group www-data and change default wp:file permissions from chmod:644 files 2.5 directories to chmod:664 for files and chmod:775 for directories This way new file will be the user not www-data but editing/uploading exising files will remain www-data:www-data wp:check file permissions

15.10.7. Links

15.10.7.1. Example
# Hard links
# reference a file in the filesystem
# do not break if file is moved or deleted (hardlink will still hold the file)
ln filetolink hardlink

# hardlink has the same size as filetolink. Good to directly open or edit it
cat hardlink

# ls -al hardlink
# -rw-r--r--@

# Symbolic link or Symlink or sym link, ~abc~ at user home folder, point it to existing folder ~xyz~ at user home folder. Reference a file path or a directory path
# break if file is moved or deleted
# ln -s filetolink symlink
ln -s ~/xyz ~/abc

# ls -al ~/xyz
# lrwxr-xr-x

# Relative Path :: you are at ~/abc/xyz, you want to create ~/abc/xyz/sym to point to existing folder ~/abc/efg/source
ln -s ../eft/source/ sym

# remove the symlink, not the files of it
rm ~/abc/xyz/sym

# Directly go to the actual folder by calling ~/abc
cd -P ~/abc
  1. Move a directory to a different directory and make the original directory as a symlink to the new directory
    # Check permission
ll /path/to/source

# Create the new directory with the same permission as the source
mkdir -m 777 /path/to/destination

# Make a copy of the source
cp -a /path/to/source /path/to/source_backup
mv /path/to/source/* /path/to/destination

# remove the source directory
rm -rf /path/to/source

# create symlink
ln -s /path/to/destination /path/to/source
15.10.7.2. Summary
  • Can create symlink to non-existing file
  • Options
    -s
    symbolic link instead of hard link
    -f, –force
    remove existing destination files
  • On Windows
    • See git:symlink
    • To create a symlink on Windows, enable Developer Mode first and command prompt not PowerShell
      • mklink symlinkgoesto myfolder\mytargetfolder
      • del symlinkgoesto
  • On Vagrant, see vagrant:symlink

15.10.8. Search files or directories, sort by size

  • Return path of matched files or directories
  • ? * [] but not regex
    • used in -name -iname -path

  • -cmin 

    # last 5 minute
find . -cmin -5

# Last day 1440, Last 31 days 44640

  • -exec 

    # {} is each result of find.
# \; denotes the end of one command
find . -size +100M -exec ls -lh {} \;

# find and delete
find . -type f -name "file-to-find" -exec rm -f {} \;

# Multiple. The first one needs to return success in order for the next one to be run
find . -name "*.txt" -exec echo {} \; -exec grep banana {} \;

# Multiple. Add `-o` to make sure both commands are run
find . -name "*.txt" \( -exec echo {} \; -o -exec true \; \) -exec grep banana {} \;

# Plus sign
find . -exec ls '{}' \;
# executes:
# ls file1
# ls file2

find . -exec ls '{}' \+
# executes
# ls file1 file2

  • Get top 10 largest files in current folder 

    find . -type f -exec du -Sh {} + | sort -rh | head -n 10

  • Depth Assume it's at home folder, and here's the structure 

    ~/1/2/3
~/a/1/2/3
~/afile

    At home folder, find . will return 

    ./1
./a
/afile

    The depth of ./1 and ./a is 1 While at home folder, find ~/ will return 

    /home/user/1
/home/user/a
/home/user/afile

    The depth of /home/user/1 and /home/user/a related to ~ is 1 To find any folder that is named 1 but not at first level 

    find ~ -mindepth 1 -iname 1

  • linux:wc -not -path "a/path/*" 

    # Exclude abc/.git directory
find abc/ -type f -not -path "abc/.git/*" | wc -l

# If abc is the current directory
find . -type f -not -path "./.git/*" | wc -l

# Count for files and folders
find . | wc -l

  • -not -iname 

    find . -type f -not -name '*.jpg' -not -name '*.pdf'
find wp-content/uploads -type f -not \( -name '*.jpg' -o -name '*.pdf' -o -name '*.gif' -o -name '*.png' \)
  • wp:check file permissions

  • Sort by modified date newest first 

    find . -printf "%T@ %Tc %p\n" | sort -rn

  • -iname --exec 

    find -iname '*.jpg' -exec mv -t path/to/target/folder/ {} \+

find -iname '*.jpg' -exec cp {} path/to/target/folder/ \;

  • -perm 

    # -perm mode :: exact match
# rw-
# rw-
# r--
find . -perm 664

# -perm -mode :: match any of these
# rw*
# rw*
# r**
find . -perm -664

# -perm /mode :: Any of the permission bits mode are set for the file.

# These are the same
# Files which are writable by either their owner or their group.
# The files don't have to be writable by both the owner and group to be matched; either will do.

# 220 is
# -w-
# -w-
# ---

find . -perm /220
find . -perm /u+w,g+w
find . -perm /u=w,g=w

15.10.9. Search Text in Files

Regex is used. To match literal string, use -F
grep -Fr 'hello' .
(no term)

Basic Regular Expression 

^ $ . [ ] - and * as Meta characters

\d # any digit
\D # anything not a digit
\w # any word (alphanumeric + underscore)
\W # anything not a word character
\s # whitespace (space, tab, line break)
\S # anything not whitespace
Character Class
anything inside a pair of brackets []
  • Special character classes

  • https://www.regular-expressions.info/posixbrackets.html#class 

    echo 'AaBbCcDdEe' | grep --color '[[:upper:]]'
    • [:alpha:]
    • [:digit:]
    • [:alnum:]
    • [:lower:]
    • [:upper:]
    • not including spaces. ASCII eq. [!"\#$%&'()*+,\-./:;<=>?@\[\\\]^_‘{|}~]
    • space character (space, tab, new line)
    • whitespace character
    • printable characters, including space
    • printable characters, not including space
    • Control character (non-printing)
    • Hexadecimal characters (0-9, A-F, a-f)
  • Refer to js:regex:character class
(no term)

Basic usage 

# only files at this directory, not subdirectories
grep 'abc' /root/dir

# recursive (subdirectories as well)
grep -r 'abc' /root/dir
# for current directory
grep -r 'abc' .

grep --color -HRi "google\.com" /root/dir

# Only file names
grep --color -HRi "abc" /root/dir | cut -d: -f1

# Unique file names
grep --color -HRi "abc" /root/dir | cut -d: -f1 | sort -u

# Exclude file types
grep --color -HRi --exclude=\*.{pdf,jpg,jpeg,JPG,mp3,png,bmp,sql} "abc" .

# Exclude binary files
grep --color -HIRin --exclude=\*.{pdf,jpg,jpeg,JPG,mp3,png,bmp,sql} "abc" .

# Return 5 trailing lines and 10 leading lines for each match
grep -A 5 -B 10 prefork.c /etc/httpd/conf/httpd/conf
(no term)

Extended Regular Expression (, ), {, }, ?, + and | as extra Meta characters 

grep -E 'o' geeks.txt

# 2 or more vowels
grep -E '[aeiou]{2,}' geeks.txt

# 1 to 2 vowels
grep -E '[aeiou]{1,2}' geeks.txt

# exactly 2 vowels
grep -E 'el{2}' geeks.txt

# Start of a word
grep -E -i '\<h' geeks.txt

# End of a a word
grep -E 'y\>' geeks.txt

# Word boundary :: match a whole word
grep -E '\bGlenn\b' geeks.txt

# Word boundary :: `way` must be part of a word or the whole word
grep -E '\Bway\B' geeks.txt

# OR
ls -al | grep -i -E 'folder1$|folder2$'
(no term)
Options
–color
hightlight
-H, -h
return only the file names
-R, -r, --resursive
recursive
-w
whole word
  • grep -w apple fruit.txt
-v
inversion. Return lines that don't match
  • grep -v apple fruit.txt
-L
like -v, suppress normal output, but print names of files that don't have a match
-l
like -L but print names of files that have a match
-i
ignore case
-I
process binary files as if they don't contain matching data
-n
show line number
-c
show count only
-A <NUM>
number of trailing lines of text
-B <NUM>
number of leading lines of text
-F
literal text match instead of using regex by default
-E
extended regex
(no term)
--exclude-dir=node_modules
(no term)
GREP_OPTIONS
  • Set default options using linux:env
  • export GREP_OPTIONS='--color=auto --binary-files=without-match' then you don't have to provide -I and --color in command lines
  • Escape single quotes and backslash using backslash and separate options by space.

15.10.10. Translate

Syntax
tr [OPTIONS]... SET1 [SET2]
(no term)
Replace SET1 with SET2
echo '123456789' | tr '12345678' 'ABCDEFGH'
result ABCDEFGH9
Change to uppercase
echo 'hello world!' | tr [a-z] [A-Z]
  • eq. to tr '[:upper:]' '[:lower:]'
(no term)

Delete with -d 

echo 'abc123' | tr -d [:digit:]
# abc

echo 'abc123' | tr -dc [:digit:]
# 123
# c option means NOT! inverse, contradict

echo 'abc1233deee567f' | tr -s [:digit:]
# abc123deeee567f
# -s is squeeze consecutive matched characters

echo 'abc1233deee567f' | tr -sc [:digit:]
# abc1233de567
# c inverse again

echo 'abc1233deee567f' | tr -ds [:digit:] [:alpha:]
# abcdef
# delete digits, squeeze on alpha characters

echo 'abc1233deee567f' | tr -dsc [:digit:] [:digit:]
# 123567
# c only applies to d here..

tr -d '\015\032' < windows_file > unix_file

15.10.11. column bash:column

Default delimiter is space

Use colon
cat /etc/passwd | column -t -s ':'
Use tab
cat atabdelimited.csv | column -t -s $'\t' or column -t -s '\t' Refer to linux:/etc/passwd

15.10.12. cut

  • Extract columns from file
  • cut OPTION... [FILE]...
  • ">field 2 with delimiter tab
  • Options
    -d $'\t'
    tab delimiter. It is also the default when -f is used
    -f 2
    field 2 after delimited
    -c 2-10,72-
    each line, grab character from 2nd to 10th position and 72 to the end. Index starts from 1

15.10.13. awk bash:awk

  • https://www.tutorialspoint.com/awk/index.htm
  • Space delimited by default
  • Refer to bash:read
  • awk 'BEGIN {command1; command2;} /pattern/ {command3; command4;} END {command5; command6;}'
    • BEGIN run AWK commands 1 and 2 before read
    • Read a line from input then execute AWK commands 3 and 4 using the line (/pattern/ to filter the line)
    • repeat
    • END run AWK commands 5 and 6
awk 'BEGIN{printf "Col1\tCol2\tCol3\n"} {print}' marks.txt

# use a command file
awk -f command.awk marks.txt

# command.awk content
# BEGIN{printf "Col1\tCol2\tCol3\n"} {print}

# define variables inside awk
awk -v name=Jerry 'BEGIN{printf "Name = %s\n", name}'

# pass variable to awk
dockercontainer=lucee
awk -v name="$dockercontainer" 'BEGIN {print name}'
awk -v name="$dockercontainer" '{ if ( $1 == name ) print $2}'

# print columns
awk '{print $3 "\t" $4}' marks.txt

# Using pattern to filter lines to run body commands
awk '/somepattern/ {print $3 "\t" $4}' marks.txt

# line count
awk '/somepattern/ {print $3 "\t" $4; ++cnt } END {print "Count = ", cnt}' marks.txt

# $0 is the line, length($0)
awk 'length($0) > 18 {print length($0)}' marks.txt
15.10.13.1. Options
-F
change delimiter. Default is space
tab
-F $'\t' or -F "\t"
colon or comma
-F ":" -F ','
(no term)
Separator can be a single character or a regex
(no term)
Default space, multiple spaces are treated as one column
Other single character
e.g. for comma, multiple commas are treated as multiple columns
(no term)
To enable multiple commas as one comma/column, use -F ',*'
15.10.13.2. AWK variables
Print 17th column
cat access.log | awk {print $17}
(no term)
Print all variables to awkvars.out
  • awk --dump-variables '' with no commands
  • Print one var awk 'BEGIN {print "Arguments =", ARGC}' f1.txt f2.txt
(no term)
May change variables using -v FS='\t' or BEGIN {FS= ","}
(no term)
ARGC and ARGIND
  • multiple files as input
    • awk 'BEGIN {print "Arguments =", ARGC}' f1.txt f2.txt number of arguments 3 which are awk, f1.txt and f2.txt
    • only 1 argument awk
  • ARGIND is the index print "Filename = ", ARGV[ARGIND]
  • multiline commands do not require \
awk 'BEGIN { 
   for (i = 0; i < ARGC - 1; ++i) { 
  printf "ARGV[%d] = %s\n", i, ARGV[i] 
   } 
}' f1.txt f2.txt
  • FILENAME
  • input field separator. Default space
  • instead of using FS, a space-delimited list of field widths variable can be set to define fields
  • output field separator. Default space
  • output record separator. Default \n. This is used in print in e.g. command 3 and 4. Manually add \n for printf "%g/n",$1
  • record separator. Default newline \n
  • number of fields/columns in current line
    • Print 3rd column to the last column
      • awk '{for(i=2; i<=NF; ++i) printf "%s ", $i; print ""}'
      • awk '{for(i=3; i<=NF; ++i){printf("%s ", $i)}; printf("\n");}'
  • the number of the current line
  • similar to NR but relative to the current file
  • when it's set, pattern becomes case-insensitive
    • awk 'BEGIN{IGNORECASE = 1} /amit/' marks.txt

Variables that are functions related

RSTART
the first position in the string matched by match function awk 'BEGIN { if (match("One Two Three", "Thre")) { print RSTART } }'
15.10.13.3. AWK operators
awk 'BEGIN { a = 50; b = 20; print "(a + b) = ", (a + b) }'

# exponential
awk 'BEGIN { a = 10; a = a**2; print "a =", a }'
# or
awk 'BEGIN { a = 10; a = a ^ 2; print "a =", a }'

# result a and b are equal to 11
awk 'BEGIN { a = 10; b = ++a; printf "a = %d, b = %d\n", a, b }'

# a = 11, b = 10
awk 'BEGIN { a = 10; b = a++; printf "a = %d, b = %d\n", a, b }'

awk 'BEGIN { cnt = 10; cnt += 10; print "Counter =", cnt }'

awk 'BEGIN { cnt = 10; cnt *= 10; print "Counter =", cnt }'

awk 'BEGIN { cnt = 100; cnt /= 5; print "Counter =", cnt }'

awk 'BEGIN { cnt = 100; cnt %= 8; print "Counter =", cnt }'

# exponential
awk 'BEGIN { cnt = 2; cnt **= 4; print "Counter =", cnt }'

awk 'BEGIN { a = 10; b = 10; if (a == b) print "a == b" }'

# != < <= >=

# && || !

# ternary
awk 'BEGIN { a = 10; b = 20; (a > b) ? max = a : max = b; print "Max =", max}'

# unary
# a = 10
awk 'BEGIN { a = -10; a = -a; print "a =", a }'

# a = -10
awk 'BEGIN { a = -10; a = +a; print "a =", a }'

# string concatenation
awk 'BEGIN { str1 = "Hello, "; str2 = "World"; str3 = str1 str2; print str3 }'

# regex
# if line contains 9
awk '$0 ~ 9' marks.txt

# if line does not contain 9
awk '$0 !~ 9' marks.txt
15.10.13.4. AWK functions
# arithmetic
# atan2(y,x)
# cos(expr)
# sin(expr)
# exp(expr)
# int(expr) :: truncate to an integer value
# log(expr)
# rand()
# srand([expr]) :: random number using seed value
# sqrt(expr)

awk 'BEGIN {
   PI = 3.14159265
   x = -10
   y = 10
   result = atan2 (y,x) * 180 / PI;

   printf "The arc tangent for (x=%f, y=%f) is %f degrees\n", x, y, result
}'

# sort array based on values and change indexes to sequential integers starting with 1
awk 'BEGIN {
   arr[0] = "Three"
   arr[1] = "One"
   arr[2] = "Two"
   print "Array elements before sorting:"

   for (i in arr) {
  print arr[i]
   }
   asort(arr)
   print "Array elements after sorting:"

   for (i in arr) {
  print arr[i]
   }
}'

# sort array by indexes, after sort, the array value is the original index

# gsub(regex, sub, string)
# if string is omitted, $0 (current line) is used

awk 'BEGIN {
   str = "Hello, World"
   print "String before replacement = " str

   gsub("World", "Jerry", str)
   print "String after replacement = " str
}'

# if string contains, the first occurance position is 1. Not contain is 0
awk 'BEGIN {
   str = "One Two Three"
   subs = "Two"
   ret = index(str, subs)

   printf "Substring \"%s\" found at %d location.\n", subs, ret
}'

# length(str)

# match(str, regex)
# returns the index of the first longest match of regex in string str. Return 0 if no match found

awk 'BEGIN {
   str = "One Two Three"
   subs = "Two"
   ret = match(str, subs)

   printf "Substring \"%s\" found at %d location.\n", subs, ret
}'

# splits the string str into fields by regular expression regex and the fields are loaded into the array arr. If regex is omitted, then FS is used.
awk 'BEGIN {
   str = "One,Two,Three,Four"
   split(str, arr, ",")
   print "Array contains following values"

   for (i in arr) {
  print arr[i]
   }
}'

# Decimal num = 123
# Octal num = 83
# Hexadecimal num = 291
awk 'BEGIN {
   print "Decimal num = " strtonum("123")
   print "Octal num = " strtonum("0123")
   print "Hexadecimal num = " strtonum("0x123")
}'

# one time substitution
# sub(regex, sub, string)
awk 'BEGIN {
   str = "Hello, World"
   print "String before replacement = " str

   sub("World", "Jerry", str)
   print "String after replacement = " str
}'

# substr(str, start, l)
# returns the substring of string str, starting at index start of length l. If length is omitted, the suffix of str starting at index start is returned.

# tolower(str)
# toupper(str)
15.10.13.5. AWK pretty print
# horizontal tab
awk 'BEGIN { printf "Sr No\tName\tSub\tMarks\n" }'

# vertical tab (like IDE uses TAB to represent code hierarchy
# Parent > child > grand child > grand grand child
awk 'BEGIN { printf "Sr No\vName\vSub\vMarks\n" }'

# backspace
# Field Field Field Field 4
awk 'BEGIN { printf "Field 1\bField 2\bField 3\bField 4\n" }'

# carriage return /r or ASCII value 13 or 0x0D
awk 'BEGIN { printf "Field 1\rField 2\rField 3\rField 4\n" }'

# form feed (page breaker) \f or ASCII value 12 or 0x0C
awk 'BEGIN { printf "Sr No\fName\fSub\fMarks\n" }'

# first character, if value is numeric, ASCII convert to a character. If string, output the 1st character
# ASCII value 65 = character A
awk 'BEGIN { printf "ASCII value 65 = character %c\n", 65 }'

# %d and %i :: only print the integer part of a decimal number
# Percentags = 80
awk 'BEGIN { printf "Percentags = %d\n", 80.66 }'

# %e and %E :: [-]d.dddddde[+-]dd
# Percentags = 8.066000e+01
# %E is
# Percentags = 8.066000E+01
awk 'BEGIN { printf "Percentags = %E\n", 80.66 }'

# %f :: [-]ddd.dddddd
# Percentags = 80.660000
awk 'BEGIN { printf "Percentags = %f\n", 80.66 }'

# %g :: use %e or %f, whichever is shorter
# %G :: use %E or %f, whichever is shorter
awk 'BEGIN { printf "Percentags = %g\n", 80.66 }'
awk 'BEGIN { printf "Percentags = %g\n", "80.66%" }'

# %o :: unsigned octal number
# %u :: unsigned decimal number
# %x :: unsigned hexadecimal number. %X uses uppercase

# %% :: escape % and prints %

# %10d :: 10 characters long
# %05d :: with leading zeros and total length 5 characters long

# %-5d :: When value length is less than 5, add spaces to the right so that the value is left adjusted.

# %+d :: always add prefix to indicate whether number is positive or negative
15.10.13.6. AWK output redirection
# overwrite
awk 'BEGIN { print "Hello, World !!!" > "/tmp/message.txt" }'

# append
awk 'BEGIN { print "Hello, World !!!" >> "/tmp/message.txt" }'

# pipe
awk 'BEGIN { print "hello, world !!!" | "tr [a-z] [A-Z]" }'

# 2-way communication
# don't understand yet...
15.10.13.7. AWK array
awk 'BEGIN {
   fruits["mango"] = "yellow";
   fruits["orange"] = "orange";
   fruits["pear"] = "pear";
   delete fruits["pear"];
   # multi dimensional (just use "...")
   md["0,0"] = 100;
   md["0,1"] = 200;
   print fruits["orange"] "\n" fruits["mango"];
}'

Array loop 

awk 'BEGIN { 
   arr[0] = 1; arr[1] = 2; arr[2] = 3; for (i in arr) printf "arr[%d] = %d\n", i, arr[i]
}'
15.10.13.8. AWK if
awk 'BEGIN {
   a = 30;

   if (a==10)
     print "a = 10";
   else if (a == 20)
     print "a = 20";
   else if (a == 30)
     print "a = 30";

   if (a==20) {
     print "1";
     print "2";
   }
}'
15.10.13.9. AWK loops
# for loop
awk 'BEGIN { for (i = 1; i <= 5; ++i) print i }'

# while loop
awk 'BEGIN {i = 1; while (i < 6) { print i; ++i } }'

# do-while loop
awk 'BEGIN {i = 1; do { print i; ++i } while (i < 6) }'

# break
awk 'BEGIN {
   sum = 0; for (i = 0; i < 20; ++i) { 
  sum += i; if (sum > 50) break; else print "Sum =", sum 
   }
}'

# continue
awk 'BEGIN {
   for (i = 1; i <= 20; ++i) {
  if (i % 2 == 0) print i ; else continue
   } 
}'

# exit
awk 'BEGIN {
   sum = 0; for (i = 0; i < 20; ++i) {
  sum += i; if (sum > 50) exit(10); else print "Sum =", sum 
   } 
}'

15.10.14. Folder structure 

which tree
# apt-get install tree
# Mac: brew install tree

# list only folders without files
tree -d test/

# list everything
tree test/

# List first level child folder only for the current folder
tree -d -L 1

# List hidden files also
# In no event, tree prints the file system constructs `.`  (current  directory) and  `..` (previous directory).
tree -a /path/to/dir

15.10.15. Compare files in 2 directories 

diff --brief -Nr dir1/ dir2/
# -N shows difference of file existance

diff -r dir1/ dir2/
# shows text comparison as well

# compare filenames and directory names only
diff <(cd dist && find . | sort) <(cd src && find . | sort)

# if dist has test.txt but src doesn't, it will show
# < ./test.txt

15.10.16. Compare text files 

diff file1.txt file2.txt

# Header
# 3d2 :: line 3 on the left file, line 2 on the right file
# - d :: deletion
# - a :: adding
# - c :: changing

# Body under Header
# < means the line belongs to the left file
# > means the line belongs to the right file

diff -u file1.txt file2.txt
# unified diff, which shows a bit differently

# Use colordiff, it's better!
  • Options
    -i
    case insensitive
    -b
    ignore chagnes to blank characters
    -w
    ignore all whitespace
    -B
    ignore blank lines
    -r
    recursive for directories
    -s
    show identical files
  • Output options
    -c
    copied context
    -u
    unified context
    -y
    side by side
    -q
    only whether files differ

15.10.17. rm

  • Options
    -f, --force
    silent
15.10.17.1. Delete using brace expansion
rm -rf abc.log.2012-03-{14,27,28}
# eq.
rm -rf abc.log.2012-03-14 abc.log.2012-03-27 abc.log.2012-03-28
15.10.17.2. Delete all files and sub directories
# delete the dir as a whole
rm -rf /path/to/dir

# delete all non-hidden files and sub directories in the dir but keep dir
rm -rf /path/to/dir/*

# delete files only and keep sub directories in the dir
find /path/to/dir -type f -delete
15.10.17.3. Delete folders with a name
# Test it first
find ~/www -iname WEB-INF -exec ls {} \;

# Delete files in those folders
find ~/www -iname WEB-INF -exec rm -rf {} \;
15.10.17.4. Delete all files and subdirectories of the current directory
find . -mindepth 1 -delete

# -mindepth 1 is to leave the directory itself alone

15.10.18. ftp bash:ftp

15.10.18.1. Basics
ftp domain.com
ftp 192.168.0.1
ftp user@ftpdomain.com

# you get a prompt
# ftp>

# Commands
ls
cd afolder

# set local directory where downloaded files will be stored on your local environment (not the ftp server)
lcd /home/user/yourname

# download a file
get path/on/ftp/to/file

# download multiple files
mget *.xls

# for download, better use bash:wget

# upload a file that is in local folder
put filename

# upload a file that is in any folder on your local environment
put /home/user/your/path/to/a/file

# upload multiple files
mput *.xls

# close ftp connection
bye
exit
quit

# for help
help
15.10.18.2. lftp

To enable wildcard for rm, use glob 

#!/bin/bash
ftpsite="ftp://yourftp.com"
ftpuser="b"
# password doesn't seem to have to be escaped like Makefile
ftppass="c"

# make sure you are sure which folder to delete before run mdelete and rmdir
# to delete ~/target/path/targetfolder

lftp u $ftpuser,$ftppass $ftpsite -e 'set ssl:check-hostname false;set ftp:list-options -a;' <<EOMYF
cd target/path
ls -d */ .*/
ls -d targetfolder/
rm -r targetfolder
rm -r targetfolder2
rm -r targetfolder3
glob -a rm -r ./*images*
quit
EOMYF

exit 0
15.10.18.3. Options for ftp command
-i
turn off interactive prompting during multiple file transfers
-n
restrain from attempting auto-login upon initial connection

15.10.19. cp

  • Basics 

    # Copy a sub folder abc/ in the current directory to a new directory ~/xyz with everything preserved
cp -a abc/ ~/xyz

# Duplicate subfolder abc and name it xyz
cp -a abc xyz

# Copy multiple files to a directory
cd /path/to/files
cp -a 1.php 2.php /move/to/this/directory

# Copy files and subdirectory of the source directory to the target directory
cp -a /path/to/source/directory/. /path/to/target/directory/
  • Options
    -a
    -dR --preserve=all
    -d
    --no-dereference --preserve=links
    -P, --no-dereference
    never follow symbolic links in SOURCE
    -R, -r, --recursive
    for directory
    -i
    default it will overwrite. Interactive to ask me
    -f, --force
    if an existing destination file cannot be opened, remove it and try again
  • Copy the same file to multiple destinations use linux:tee

    • No need to create destination files 

      tee ~/dest1/a.txt ~/dest2/a.txt < ~/source/a.txt > /dev/null

  • To copy and exclude some directories use rsync 

    # At parent folder which is the parent for sourceFolder and destinationFolder
rsync -Pavin sourceFolder/ destinationFolder --exclude node_modules

# remove -n to actually run rsync

  • Copy and rename ABC.* to DEF.* 

    # test, return cp commands
# for file in ABC.*; do echo cp -a "$file" "${file/ABC/DEF}";done

for file in ABC.*; do cp -a "$file" "${file/ABC/DEF}";done

15.10.20. mv

# rename a file
mv afile.txt afile2.txt

# rename a directory
mv path/to/dir path/to/nonexsitingdir

# move a directory to another existing directory so that path/to/existingdir/dir
mv path/to/dir path/to/existingdir
# eq.
mv path/to/dir path/to/existingdir/

# move a directory to the current directory so that path/to/destination_dir/dir
cd path/to/destination_dir
mv path/to/dir .

# move a directory's files and subdirectories to another existing directory so that path/to/existingdir
# path/to/dir remains
mv path/to/dir path/to/existingdir

# move a subdirectory's files and subdirectories to its parent directory
mv -f subdirectory/{.,}* .

# no overwriting
mv -n path/to/dir path/to/existingdir

# forece overwriting. This is the default!!
mv -f path/to/dir path/to/existingdir

# interactive asking for overwriting
mv -i path/to/dir path/to/existingdir

15.10.21. mount

15.10.21.1. mount --bind
# Bind `source` directory to `destination` directory
# `destination` directory content is set to the same as `source` directory
# - Symlink is not created
mount --bind path/to/source path/to/destination

# See if a directory is a mountpoint
# 1. path/to/destination is a mountpoint
mountpoint path/to/destination
# 2. `Mounted on` shows path/to/destination.
#    For non-mountpoint directory, `/` is shown
df path/to/destination

# Show the source directory the mountpoint points to
findmnt path/to/destination

# Unmount to remove the bind
umount path/to/destination

15.10.22. tar

15.10.22.1. Compress and Read .tar file
# /home/targetFolder
cd /home
tar -cvzpf backup.tar.gz targetFolder

# Read .tar file
tar -tf backup.tar
# Read .tar.gz file
tar -tzf backup.tar.gz

# Count number of files
tar -tzf backup.tar.gz | wc -l

# targetFolder/index.html
# targetFolder/subfolder/index.html
# ...

# Exclude a folder
tar -cvzpf backup.tar.gz targetFolder --exclude='wp-content/uploads' --exclude='anotherFolder'

# Compress the current folder and save the archive in current folder
cd /home/targetFolder
tar -cvzpf backup.tar.gz .

# Compress the current folder and save the archive in one level above
tar -cvzpf ../backup.tar.gz .

# The archive structure is
# ./index.html
# ./subfolder/index.html
# ...

Options

-z
use gzip method
-c
create a new archive
-C
current directory
-v
verbose
–list, -t
List the contents
-p
preserve permissions (default for superuser)
-f
use archive file as result or input. It follows the output/input filename
–exclude
wrap in '' if it contains spaces
–exclude='*'
ignore 3 levels deep

Say you want compress the html folder at var/www/html, do this 

tar -cvzpf mybk.tar.gz -C /var/www/html .
# tar -tzf mybk.tar.gz
#  ./fileone ./filetwo

# instead of
tar -cvzpf mybk.tar.gz /var/www/html
# tar -tzf mybk.tar.gz
# /var/www/html/fileone /var/www/html/filetwo

tar -cvzpf /path/to/mybk.tar.gz -C /var/www/html fileone
# tar -tzf /path/to/mybk.tar.gz
# ./fileone

Dry run :: use - for tar file name and pipe with wc 

tar -cvzpf - /var/www/html | wc -c
15.10.22.2. Uncompress
# Read the tar file first
tar -tf backup.tar.gz

# If file structure is
# targetFolder/index.html
# targetFolder/subfolder.html ...
# And put targetFolder so that /anotherfolder/targetFolder
# Copy .tar.gz to /anotherfolder
cd /anotherfolder
tar -xvzf backup.tar.gz

# extract only some files, only one `--wildcards` at a time
tar -xvzf bk.tar.gz --wildcards '*.png' --ignore-case

# If you don't want to copy .tar.gz to /anotherfolder, 
tar -xvzf /any/path/to/backup.tar.gz -C /path/to/anotherfolder/

# if you want to untar targetFolder only in the tar file
tar -xvzf /any/path/to/backup.tar.gz targetFolder

# if you want to untar targetFolder only and save the uncompressed as /anotherfolder/* not /anotherfolder/targetFolder
tar -xvzf /any/path/to/backup.tar.gz --strip-components=1

# If file structure is
# ./index.html
# ./subfolder/index.html ...
# Then copy backup.tar.gz to /anotherfolder/targetFolder
cd /anotherfolder/targetFolder
tar -xvzf backup.tar.gz

# tar -xvzf /any/path/to/backup.tag.gz -C /anotherFolder/targetFolder

# Uncompress file.gz, not file.tar.gz, 
gzip -d file.gz 

# The gz file will disappear

Options

-x
extract
-C
change the folder that the tar file will be uncompressed to. Without it, it will be the current folder

15.10.23. rsync

15.10.23.1. Basic
  • Both client and server must have rsync installed
    • rsync -v
  • rsync [OPTION...] SRC... [DEST]
    SRC
    If SRC ends with /, it means only the files and subdirectories in that SRC directory. If it doesn't end with / and the SRC is a directory, then the whole directory is the SRC
    DEST
    it doesn't matter to end the DEST directory with or without /
  • rsync -rlvzuPn --size-only $SOURCE $DESTINATION
    -n
    dry run
    -z, --compress
    compress data sent to destination machine
    -u, --update
    skip files that are newer in $DESTINATION
    • See --ignore-existing
  • Verbose and output
    -v
    verbose
    --stats
    show file-transfer stats: number of files, number of files transferred, Total file size in bytes, Total transferred file size
    -i
    Very useful! Display change-summary for all updates, also use with --stats
    -P
    --progress --partial
    • show progress and to resume interrupted transfers
  • -rlptgoD
    • recursive, symlink, same permissions, mod time, group/owner, device/special files
    • recursive
    • symlink
    • preserve permissions. The receiving rsync sets the destinaion permissions to be the same as the source permissions
      • --no-p
    • actually is very crucial. This tells rsync to transfer modification times along with the files and update them on the remote system. Unless --size-only is used, missing -t will cause the next transfer to behave as if it used -I, causing all files to be updated
      • This command will not sync files with different permissions, ownership and modification time. Review the results and remove dry run -n before running again
      • By default, rsync finds files that need to be transferred using a "quick check" algorithm that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated
      • This modifies rsync’s "quick check" algorithm for finding files that need to be transferred, changing it from the default of transferring files with either a changed size or a changed last-modified time to just looking for files that have changed in size. This is useful when starting to use rsync after using another mirroring system which may not preserve timestamps exactly
    • Use it with --size-only. Refer to pantheon:rsync
      -I, --ignore-times
      Normally rsync will skip any files that are already the same size and have the same modification timestamp. This option turns off this “quick check” behavior, causing all files to be updated
    • To set the group of the destination file to be the same as the source file. Only if dest is run as super-user
      • --no-g
    • To set the owner of the destination file to be the same as the source file. Only if dest is run as super-user
      • --no-o
    • To give new files the destination-default permissions (while leaving existing files unchanged), make sure -p is off and use --chmod=ugo=rwX
      • rsync -zar --no-p --no-g --no-o --chmod=ugo=rwX
    • to use the default permissions of the destination dir
      --no-g
      to use the default group of the destination dir
      --no-o
      to use the default user of the destination dir
    • hard set owner:group on dest. rsync -rlvzPitog --chown=www-data:www-data /source /destination
    • is equivalent to --devices --specials
      --devices
      To transfer character and block device files to the remote system to recreate these devices
      --specials
      To transfer special files such as named sockets and fifos
  • exclude a file or directory. Multiple --exclude can be used
    --exclude='*.log'
    by file extension

  • exclude multiple files and folders, use relative path without leading ./ 

    sources
public_html/database.*
downloads/test/*
  • Only include log files --include='*.log' --exclude='*'
  • skip updating files that already exist on the destination
    --ignore-non-existing, --existing
    skip creating files and directories that do not exist yet on the destination
  • By default, rsync uses ~/.ssh/config on local to SSH into remote. Change the default e.g.
    • -e 'ssh -p 2234'
    • -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'
    • See bash:sshpass
  • On remote, specify which program (default rsync). e.g. change it to sudo rsync to elevate permissions
  • Dangerous!
    --delete
    Delete extra files from the receiving side (ones that aren't on the sending side), but only for the directories that are being synchronnized. By default, rsync does not delete files on the receiving side
15.10.23.2. Change summary, compare 2 directories

For comparing, you don't need -a specifically -pgoD so that left -rlt You need -vzPin Probably include -t and –size-only

rsync -rltvzPin wp-content/themes/my-theme/ sshconfighostname:/var/www/mylivesite/wp-content/themes/my-theme/

For current folder, use ./

Refer to bash:sshpass 

.d..t...... ./
<f.st...... footer.php
<f.st...... front-page.php
.f..t...... functions.php
<f.st...... header.php
.f..t...... index.php
.f..t...... page--coupon-image_tpl.php
.f..t...... page--coupons_tpl.php
.f..t...... style.css
.d..t...... lb/
cd+++++++++ lb/api-localbusiness/
<f+++++++++ lb/api-localbusiness/cf-form-post.php
<f+++++++++ lb/api-localbusiness/cf-form-post_rel0.php
<f+++++++++ lb/api-localbusiness/cf-form-post_rel2.php
.d..t...... lb/css/
.f..t...... lb/css/font.css
<f.st...... lb/css/style.css
.d..t...... lb/font/
.d..t...... lb/font/trade-gothic/
.f..t...... lb/font/trade-gothic/bold.eot
.f..t...... lb/font/trade-gothic/bold.otf
.d..t...... lb/img/
.f..t...... lb/img/alectra-logo.eps
.d..t...... lb/img/bg/
.f..t...... lb/img/bg/lp-bg_001.jpg
.f..t...... lb/img/bg/lp-bg_002.jpg
.d..t...... lb/img/icons/
.f..t...... lb/img/icons/contact.png
.f..t...... lb/img/icons/social-facebook.png
.d..t...... lb/img/icons/coupon/
.f..t...... lb/img/icons/coupon/icon-01.svg
.f..t...... lb/img/icons/coupon/icon-02.svg
.d..t...... lb/img/logos/
.f..t...... lb/img/logos/brampton-hydro.png
.f..t...... lb/img/logos/enersource.png
.d..t...... lb/js/
.f..t...... lb/js/app.js
.f..t...... lb/js/bicubic-interpolation.js

YXcstpoguax

Y is update type

  • < means a file is being transferred to the remote (sent)
  • > means a file is being transferred to the local (received)
  • c means a local change/creation is occuring for the item (on the receiving side)
  • h means the item is a hard link to another item
  • . means the item is not being updated (though it might have attributes that are modified)
  • * means the rest of the itemized-output area contains a message (e.g. "deleting")

X is file-type :: f for a file, d for directory, L for symlink, D for a device, S for a special file (named sockets and fifos)

cstpoguax are attributes.

  • "." stands for no change.
  • A newly created item replaces each letter with a "+"
  • An identical item replaces the dots with spaces
  • An unknown attribute replaces each letter with a "?" (could happen when talking to an older rsync)
  • c: checksum is different
  • s: file size is different and will be updated
  • t: modification time is different and is being updated to the sender's value (requires -t or –times)
  • T: modification time will be set to the transfer time
  • p: permissions are different and are being updated to the sender's value (requires –perms)
  • o: owner is different and is being updated to the sender's value (requires –owner)
  • g: group is different and is being updated to the sender's value (requires –group)
  • u: reserved for future use
  • a: ACL information changed
  • x: Extended attribute information changed.
15.10.23.3. Filter rules (INCLUDE/EXCLUDE Patterns)

If the pattern ends with a / then it will only match a directory, not a regular file, symlink or device If the pattern starts with a / then it is similar to a leading ^ in regular expressions. lq/foorq would match at any point in the hierarchy. oq*cq matches any path component but it stops at slashes use ** to match anything, including slashes oq?cq matches any character except a slash oq[cq introduces a character class, e.g. [a-z] or 

[[:alpha:]]

A trailing *** will match both the directory and everything in the directory e.g. foldername/***

15.10.23.4. Run rsync on remote using non-root SSH credentials
  • Need to make sure the non-root SSH credentials can run sudo rsync
  • Files uploaded to remote will have the default permissions, group and user of the destination remote dir
# --rsync-path 'sudo rsync'
# --rsync-path="sudo rsync"
# "--rsync-path=sudo rsync"
# '--rsync-path=sudo rsync'
rsync ... -zar --no-p --no-g --no-o --rsync-path='sudo rsync' ...
15.10.23.5. Examples
# change owner
# - Dry, recursive, symlink, verbose, compress
# - show progress and make it possible to re-rsync if connection is lost
# - show changes in detail for each transferred file
# - update time on destination
# - don't transfer file if dest has mod time newer
# - update owner and group on dest to be the same as source
# - change owner:group on dest

rsync -rlvzPituogn --chown=www-data:www-data wp-content/ remote:path/to/wp-content

# Transfer everything from remote to local
# - verbose
# - show overall stats
# - show individual file log
# - show progress and re-sync possible
# - compress
# - files/directories are created on local using local file permission mask

rsync -avziPn --stats $(devhost):$(devhostdir)/wp-content/uploads/ wp-content/uploads/

# Simple Download from Remote
rsync -rlvzPitun --stats remote:path/to/site/sites/default/files/ sites/default/files

rsync -a /vagrant/ma/test1 --files-from=/vagrant/ma/list.txt /vagrant/ma/test2
rsync -aI /vagrant/ma/test1 --files-from=/vagrant/ma/list.txt /vagrant/ma/test2
# list.txt
# folder1/a.txt
# abc.txt

15.11. Processes

ps -ef List of running processes
sudo ls -l /proc/PID/exe file path for command, sub PID
ps -f -p 123 List file of a running process
ps aux List all processes for all users and
  not started by a terminal (e.g. started by a daemon)
  and show user columns
   

ps options

f or F
full format. F for Ubuntu
e
all processes. same as -A
p
PID. same as p and –pid
o
specify format e.g. -o pid,ppid,pgid,sid
ax
list all processes from all users that are not started by a terminal (e.g. started by a daemon, background process that is removed from jobs)
u
Displays user-oriented output. This includes the USER, PID, %CPU, %MEM, SZ, RSS, TTY, STAT, STIME, TIME, and COMMAND fields.
l
Displays a long listing having the F, S, UID, PID, PPID, C, PRI, NI, ADDR, SZ, PSS, WCHAN, TTY, TIME, and CMD fields. Note :: -l doesn't include fields from -u
w
use with -f to show in unlimited width so that PPID and others are shown

15.11.1. pgrep, pkill, linux:kill

# Search a process by file name, return PID
pgrep newcomgo

# Kill a process by file name
pkill newcomgo

15.11.2. Parent process linux:process:parent

Parent process id is PPID. PPID is 1 means parent is not found. Child process PID is 1234 

# search in all processes
ps -feww 1234

# just return ppid
ps -o ppid= 1234

# return with custom format
ps -eo pid,ppid,comm | grep 1234

# search in all processes that may start by daemon
ps -axl | grep 1234

A package pstree can be installed and used pstree -s -p 2072 outputs init(1)───pulseaudio(2061)───gconf-helper(2072)

15.11.3. Run executable in background for all terminal sessions

This should be enough. If not, use bash:screen :: nohup ./bin/newcomgo &

15.11.4. jobs 

# Add ~&~ to put process list into background
(tar -cf 1.tar /home/1; tar -cf 2.tar /home/2)&

# run a script file in background
./test5.sh &

# a job number and process number are returned
# [1]+ 4867

# If terminal session is ended (SIGHUP), background processes will be stopped, too
# Use nohup to block SIGHUP signals being sent to the process
nohup ./test1.sh &

# It produeces a file `nohup.out` in the directory that the script is in which contains STDOUT and STDERR.
# No output to any terminals.

# To prevent background processes from sending error to terminal
./test5.sh 2> /dev/null &

# List processes that are running in background
jobs

# List PIDs
jobs -l

# Kill a process
kill 1234
# or
kill -KILL 1234
# above is eq. to below
kill -9 1234

# Stop a process
kill -STOP 1234

# Resume or continue a stopped process
kill -CONT 1234

# job number can be used
kill -STOP %2
kill -CONT %2

# remove or prevent a background process from being managed using `jobs` (using job number) and keep it running
disown %2

# still can see the process
ps -x | grep yourscriptname
15.11.4.1. Long running process
# Stop a long running process with C-z. A job number is returned
# [1]+ Stopped        bash message.sh

# resume the stopped process in background so that new commands can be typed and executed but the process can still STDOUT
bg %1

# bring the process to foreground
fg %1

# another example
# send process to background
scp bigfile user:remote:/root &
# bring the process to foreground so that you can login
fg %1
# after login, pause the process using C-z
# resume the paused process to background
bg %1

# check status by bringing the process to foreground
fg %1
15.11.4.2. Most recent, previously and multiple jobs
# multiple jobs
fg %1 %2
bg %1 %2

# `jobs` shows + and - beside the most recent and previously bg jobs
# target most recent `+`
bg
fg

# target previously `-`
bg -
fg -

15.11.5. top

  • First line
    • current time, system up time, number of users logged in, load average (1, 5, 15 minutes; >2 is busy).
  • Second line
    • total processes: running, sleeping, stopped and zombie (have finished but parent process hasn't responded)
  • Third line
    • CPU utilization: us (user), sy (system), ni (nice: time running niced user processes), wa (time waiting for I/O completion), hi (time spent servicing hardware interrupts), si (software interrupts), st (stolen from this vm by the hypervisor)
  • Memory usage
    • Line 1 reflects physical memory: total, used, free and buffers
    • Line 2 reflects virtual memory: total, used, free and cached
  • Detailed for each process
    PID
    The process ID of the process
    USER
    The user name of the owner of the process
    PR
    The priority of the process
    (no term)
    NI: The nice value of the process
    (no term)
    VIRT: The total amount of virtual memory used by the process
    (no term)
    RES: The amount of physical memory the process is using
    (no term)
    SHR: The amount of memory the process is sharing with other processes
    (no term)
    S: The process status (D = interruptible sleep, R = running, S = sleeping, T = traced or stopped, or Z = zombie)
    (no term)
    %CPU: The share of CPU time that the process is using
    (no term)
    %MEM: The share of available physical memory the process is using
    (no term)
    TIME+: The total CPU time the process has used since starting
    (no term)
    COMMAND: The command line name of the process (program started)
  • set sort, display more fields/columns
  • Options
    • top -n 10 -o cpu -s 3 -U lili
      • top 10, order by cpu usage, update every 3 seconds, processes are of user lili

15.11.6. Multiple commands && vs || vs ;

Commands behind ; are always run Command behind && is run only when the previous returns success Command behind || is always run So || is the opposite of && 

false; echo 'yes'
true; echo 'yes'
false && echo 'yes'
false || echo 'yes'

15.12. Memory

Total memory
grep MemTotal /proc/meminfo
More info
cat /proc/meminfo
Memory usage
free -ltm -c 2 -s 2 or just free -mh
Delay 2 seconds and repeat 2 times
-c 2 -s 2
(no term)
Options
m
in megabyte. e.g. -g
t
show Total
l
show Low and High
h
human readable
Another way
vmstat -s
(no term)
See memory usage for each process, use linux:top or htop

Before in Ubuntu it shows 6 columns and the -/+ buffers/cache row total used free shared buffers cached Mem: 7916 7645 271 99 455 1764 -/+ buffers/cache: 5426 2490 Swap: 24999 805 24194

used = total - free

Now the columns are and no -/+ buffers/cache row total used free shared buff/cache available Mem: 3553 1192 857 16 1504 2277 Swap: 3689 0 3689

used = total - free - cached - buffers

Buffers are file system metadata and cache is pages with actual contents of files or block devices. Buff/cache will be freed up for newer applications that need memory to save I/O operations.

Linux OS always uses physical memory so free is always low.

A Linux system is really low on memory if the free value in -/+ buffers/cache: line gets low which is the available column in new Linux.

15.13. Service

15.13.1. SystemV init system :: other names SysVinit

  • Characteristics
    synchronous
    things have to be executed in sequence
  • List all services (on '+', off '-', managed by Upstart '?')
    • Services defined as /etc/init/*.conf or /etc/init.d/* do not appear in this list
  • service operates on the daemon files in /etc/init or /etc/init.d which is an old init system

    • Set up an PHP script file as service 

      sudo nano /etc/init/myPHP.conf

        start on startup
  stop on shutdown

  respawn

  script
cd /path/to/the/script
php myPHP.php
  end script

15.13.2. Upstart

  • Characteristics
    • Designed to replace SystemV
      event-based initialization program
      plug and play devices
      (no term)
      asynchronous
      (no term)
      automatic restart
    • no update since 2014

15.13.3. SystemD :: System Management Daemon

  • Characteristics
    • parallel

  • List running services only with more detail 

    systemctl | grep 'php'
  • systemctl operates on /etc/systemd. File in this folder will be used first otherwise fall back to old System V init
  • Some daemon files can do restart and it might be the preferred way to interact with service. e.g. sudo /etc/init.d/jenkins restart

  • Set up an PHP script file as a system Create /etc/systemd/system/myPHP.service file or symlink to this directory https://maslosoft.com/blog/2019/07/10/running-php-script-as-a-system-service-in-ubuntu/ 

    [Unit]
Description=Web Socket Server

[Service]
WorkingDirectory=/path/to/run
ExecStart=/usr/bin/php comms_scheduled.php
Restart=always

[Install]
WantedBy=multi-user.target

    # Instruct SystemD that config was updated
sudo systemctl daemon-reload
sudo systemctl enable myPHP
sudo systemctl start myPHP
systemctl status myPHP

# Looks like simply restarting the service can enable and start the new service
systemctl restart myPHP

15.13.4. systemctl 

sudo systemctl restart myapp.service
sudo systemctl reload myapp.service
sudo systemctl reload-or-restart myapp.service

sudo systemctl enable myapp.service
# /lib/systemd/system or /etc/systemd/system
# /usr/lib/systemd/system
# /etc/systemd/system/some_target.target.wants

# enable and start. enable does not start the service in the current session

15.14. Network

All active NIC and IP addresses: ifconfig All enabled and disabled NIC: ifconfig -a View one NIC (interface): ifconfig eth0

Ethernet interface: eth0, eth1 Wireless network interface: wlan0, wlan1 Loopback interface for system internal communication: lo

Find default gateway ip route | grep default

15.14.1. /etc/network/interfaces file

For Ubuntu and Debian 

iface eth0 inet static
# only one address, call static
#  address 192.168.1.5

# netmask defines ip's within this range is considered under the same LAN
# netmask 255.255.255.0

# When this machine talks to an ip out of this LAN, a gateway is needed
# gateway 192.168.1.254

# DNS
# dns-nameservers 192.168.1.254 8.8.8.8

Set interface to dhcp 

auto eth0
iface eth0 inet dhcp

After changing interfaces file, turn off and on the interface 

sudo ifdown eth0 && sudo ifup eth0

15.14.2. DNS

Define DNS hosts

/etc/nsswitch.conf and the line starting with hosts: defines which files Linux should look for the IP of a domain 

hosts: files mdns4_minimal [NOTFOUND=return] dns

  • First, /etc/hosts file 

    127.0.0.1 localhost
127.0.1.1 your-computer-name
# 127.0.1.1 is fine when the computer is a client
# If the machine is a server and it has a static IP, should set it

  • Second, it's a list of DNS nameservers you define

    See a list of DNS nameservers
    cat /etc/resolv.conf
    Not manually changable file
    /etc/resolv.conf
    (no term)
    A package called resolvconf which handles the DNS nameservers. I guess if you change /etc/network/interfaces file to use dns-nameservers, it will add nameservers to /etc/resolv.conf

  • Commands to check DNS are run based on /etc/resolv.conf 

    getent hosts google.com
host google.com
dig google.com
    • dig can only follow through DNS management. If there's redirect done in Apache, Nginx or other codes, it won't follow through them
    • Use 15.20.1 to track further
    • dig google.com or dig google.com +short
    • dig google.com MX
    • dig google.com ANY
    • dig -x 72.30.38.140
    • DNS Propagation Checker
  • Flush DNS

15.14.3. Flush DNS

  • Ubuntu
    • sudo /etc/init.d/networking restart

15.14.4. Get Server's Public IP linux:get public ip

ip addr show eth0 | grep inet | awk '{ print $2; }' | sed 's/\/.*$//'
# or
curl http://icanhazip.com

15.14.5. CIDR Netmask (Network Mask) CIDR netmask

  • 10.0.75.5/32 = 10.0.75.5 AND 255.255.255.255 = 10.0.75.5 (a single host)
  • 10.0.75.5/24 = 10.0.75.5 AND 255.255.255.0 = 10.0.75.0 (10.0.75.5 is in the 10.0.75.0 subnet)
  • 10.0.75.5/16 = 10.0.75.5 AND 255.255.0.0 = 10.0.0.0
  • 10.0.75.5/8 = 10.0.75.5 AND 255.0.0.0 = 10.0.0.0

Private IPv4 address spaces

  • 10.0.0.0 - 10.255.255.254 = 10.0.0.0/8 (16 million number of addresses)
  • 172.16.0.0 - 172.31.255.255 = 172.16.0.0/12 (255.240.0.0) (1 million number of addresses, 172.16.x.x - 172.31.x.x)
  • 192.168.0.0 - 192.168.255.255 = 192.168.0.0/16 (65,536 number of addresses)

Internal network Class A :: 10.0.0.0 - 10.255.255.254 (10.0.0.0/8, total hosts: 16 million) Internal network Class B :: 172.16.0.1 - 172.16.255.254 (172.16.0.0/16, total hosts: 65,534) Internal network Class C :: 192.168.0.1 -192.168.0.254 (192.168.0.0/24, total hosts: 254)

15.14.6. Port

  • If a port is in use telnet localhost 9000, if it's connected, it's in use
    • It's better to use telnet IP_address port_number in an external computer to connect to a port to check if it's in use
    • To exit, first hit C-] then enter, in telnet> prompt, hit close
  • netstat
    • List all ports in use netstat -lp or netstat -nlp to list in numeric form
    • Search a port netstat -nlp | grep 8000

    • In Windows, to see if a port is being used in a local computer 

      netstat -an | find "8080"

# From outside, just telnet host port (or telnet host:port on Unix systems) to see if the connection is refused, accepted, or timeouts
    • netstat options
      l
      show only listening sockets
      p
      show PID and name of the program to which each socket belongs
      n
      show numerical addresses instead of symbolic host, port or user names
      t
      tcp
      u
      udp
  • lsof
    • List all command files (open files) that are using an Internet address (-i)
      • lsof -Pi
      • lsof -i :80
    • Options
      • -P is to print port number
    • [46][protocol][@hostname|hostaddr][:service|port]
      • 4 or 6 for IPv4 or IPv6

  • Kill processes that use the port 

    # this can also list PID's that use the port
fuser 443/tcp -v

# and kill
fuser -k 443/tcp

    If process keeps coming back, consider find the parent process and kill it

  • Port range
    • Use open ports that are in the range of 1024-49151
      • 0-1023 are well known ports
      • 1024-65535 are registered ports
      • 49152-65535 are dynamic ports and should not be prescribed to a protocol (e.g. random)

15.14.7. Wireless network adapters iwconfig

15.14.8. Gather monthly bandwidth 

sudo apt-get install vnstat
# check
vnstat
# tell it to monitor a network interface
vnstat -u -i eth0

# it will create a database for it, check what interface is being monitored
vnstat

# start daemon
sudo /etc/init.d/vnstat start

# read stats for an interface
vnstat -i eth0

# hourly, daily, monthly :: -h, -d, -m
vnstat -h -i eth0

# live traffic :: -l
# top 10 days with highest traffic :: -t

Info

tx
transmit
rx
receive

15.14.9. Ping 

# Ubuntu 18.04
apt install iputils-ping

15.14.10. Netcat 

# start listening and outputting data received on port 9006
nc -l localhost 9006
# keep listening

# open another shell to make a curl request
curl -F "textfield1=hello" -F "file1=@a.pdf" localhost:9006

# nc outputs the raw curl request
# does not work on ports used by Apache
# does not work for requests made from Postman..

15.14.11. Test email address if valid without sending

nslookup -q=mx gmail.com

You will see this: 

Server:         192.168.1.200
Address:        192.168.1.200#53

Non-authoritative answer:
gmail.com       mail exchanger = 10 alt1.gmail-smtp-in.l.google.com.
gmail.com       mail exchanger = 20 alt2.gmail-smtp-in.l.google.com.
gmail.com       mail exchanger = 5 gmail-smtp-in.l.google.com.
gmail.com       mail exchanger = 30 alt3.gmail-smtp-in.l.google.com.
gmail.com       mail exchanger = 40 alt4.gmail-smtp-in.l.google.com.

Authoritative answers can be found from:
alt1.gmail-smtp-in.l.google.com internet address = 173.194.205.26
alt1.gmail-smtp-in.l.google.com has AAAA address 2607:f8b0:400d:c02::1a
alt2.gmail-smtp-in.l.google.com internet address = 74.125.141.26
alt2.gmail-smtp-in.l.google.com has AAAA address 2607:f8b0:400c:c06::1a
gmail-smtp-in.l.google.com      internet address = 108.177.112.26
gmail-smtp-in.l.google.com      has AAAA address 2607:f8b0:4001:c12::1b
alt3.gmail-smtp-in.l.google.com internet address = 64.233.190.26
alt3.gmail-smtp-in.l.google.com has AAAA address 2800:3f0:4003:c01::1b
alt4.gmail-smtp-in.l.google.com internet address = 209.85.203.26
alt4.gmail-smtp-in.l.google.com has AAAA address 2a00:1450:400b:c03::1a

Choose any one or the root one to telnet 

telnet gmail-smtp-in.l.google.com 25

# if it's connected, it will say 220
# Then you first have to say HELO hi-whatever
HELO hi

# setup a From email, with <> brackets
mail from: <youremail@gmail.com>

# then specify the target email which you want to test
rcpt to: <test@gmail.com>

# If the target email is good, it will say 250 OK
# 550 and its message may say Recipient address rejected: User unknown in virtual alias table
quit

15.15. Firewall linux:firewall linux:ufw

Should be installed by default
sudo aptitude install ufw or sudo apt-get install ufw
Check if it's enabled
sudo ufw status verbose
Restart
sudo ufw disable and sudo ufw enable or just reload sudo ufw reload
(no term)
sudo ufw reset
(no term)
Refer to network:port to test

ufw comes with some profiles each of them is a setting for a service/app. Get all profiles sudo ufw app list 

Available applications:
  Nginx Full
  Nginx HTTP
  Nginx HTTPS
  OpenSSH

Refer to nginx:ufw for further info.

Default setting is no restriction on outgoing connection, deny all incoming connection

Allow OpenSSH app/profile first sudo ufw allow OpenSSH then sudo ufw enable enable ufw

Review setting sudo vi /etc/default/ufw

Make sure IPV6=yes in setting. Make sure DEFAULT_FORWARD_POLICY="ACCEPT" if docker is installed

Allow a service sudo ufw allow ssh Allow a port sudo ufw allow 22/tcp Allow FTP sudo ufw allow 21/tcp Allow a port range sudo ufw allow 1000:2000/tcp Allow an IP sudo ufw allow from 192.168.255.255

Allow HTTP web server sudo ufw allow 80/tcp Allow SSL/TLS sudo ufw allow 443/tcp Allow SMTP email sudo ufw allow 25/tcp

List all rules sudo ufw status numbered Delete a rule sudo ufw delete [number]

Or you can see added rules in command style sudo ufw show added (even works when ufw is inactive) Then you delete sudo ufw delete allow 22

More info

15.16. Timezone, NTP

  • Get current timezone 

    date

# show alphabetic timezone and numeric timezone
date +"%Z %z"
  • sudo dpkg-reconfigure tzdata
  • Make sure Network Time Protocol daemon (NTP) is installed which ntpd
    • Install sudo apt-get update sudo apt-get install ntp

    • Do a sync 

      sudo service ntp stop
sudo ntpd -gq
sudo service ntp start
    • Port 123 has to be open (but I think the default ufw setting is ok)
  • For Debian/Ubuntu
    cat /etc/timezone
    file stores the timezone. Default is Etc/UTC
    (no term)
    All timezones are stored in /usr/share/zoneinfo/$TZ
    • Toronto is /usr/share/zoneinfo/America/Toronto file but it's a link to Montreal file
  • See Docker > timezone for changing timezone

15.17. Font linux:font

  • Install Source Code Pro 

    git clone --depth 1 --branch release https://github.com/adobe-fonts/source-code-pro.git /usr/local/share/fonts/adobe-fonts/source-code-pro/
sudo fc-cache -fv
  • macos:font

15.18. Hardware

Most of devices: lshal

15.18.1. PCI: graphic card

# graphic card
lspci -k | grep -EA3 'VGA|3D|Display'

# more info about graphic card, get number [vendor:device]
lspci -nn

# e.g.
# 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation G96 [GeForce 9500 GT] [10de:0640] (rev a1)

sudo lspci -vvv -d 10de:0640

15.18.2. USB

USB devices
lsusb
USB 3.0
eq. to USB 3.1 Gen 1
  • 5Gbps
(no term)
USB 3.1 Gen 2
  • 10Gbps
(no term)
USB 3.2 Gen 1x1
  • Same as USB 3.1 Gen 1 and USB 3.0
(no term)
USB 3.2 Gen 1x2
  • 10Gbps
(no term)
USB 3.2 Gen 2x1
  • Same as USB 3.1 Gen 2
(no term)
USB 3.2 Gen 2x2
  • 20Gbps
(no term)
Thunderbolt is Intel and Apple only which only uses USB-C interface
  • Thunderbolt 3
    • Only 1 4K display
    • A Thunderbolt 3 upstream/hub/dock can connect to 1 other Thunderbolt 3 downstream port
  • Thunderbolt 4
    • Compatible with USB 4
    • Daisy Chain and other Thunderbolt special features
      • 2 4K displays or 1 8K display
      • A Thunderbolt 4 hub/dock can connect to 3 Thunderbolt downstream ports
    • 40Gbps

15.18.3. Monitor and Display

Test refresh rate
https://www.testufo.com/ghosting

15.19. Package, Repository

  • A software repo, also called package source, is a network server, local directory or CD/DVD
    • What software packages (versions, who package them) are available for download
  • Debian

    • All repos

      Files
      cat /etc/apt/sources.list and all files in /etc/apt/sources.list.d/
      List all repos
      grep ^ /etc/apt/sources.list /etc/apt/sources.list.d/*
      (no term)
      Format
      • Examples
        • deb http://site.example.com/debian distribution component1 component2 component3
          • deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable
        • deb-src http://site.example.com/debian distribution component1 component2 component3
      • Archive type
        deb
        contains binary packages (pre-compiled)
        deb-src
        packages in source code, Debian control file (.dsc) and the diff.gz containing the changes needed for packaging the program
      • distribution could be either
        • the release code name / alias e.g. bionic
        • the release class e.g. stable, testing
      • only include certain component(s)

      • Command to repo linux:add-apt-repository 

        sudo add-apt-repository \
 "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
      $(lsb_release -sc) \
      stable"
      • Repo URL

        • https://download.docker.com/linux/ubuntu file system structure

          • File ./dist/bionic/Release is copied to local when first time repo is registered
          • File ./dist/bionic/InRelease

          • sudo apt update compares local Release file with remote InRelease to check architectures, components, label, origin and suite (distribution)

            • To avoid checking, sudo apt-get --allow-releaseinfo-change update. Not recommended
            • Or remove the repo and add it back again. Refer to linux:ubuntu:ppa. Not recommended
            • Files under ./dist/bionic/stable/ actual files

    • Ubuntu

      • Repos / Components
      Main
      free and open-source software supported by Ubuntu team
      Universe
      free and open-source software maintained by Ubuntu community
      Backports
      newest version of pkg
      Multiverse
      not free pkg (software restricted by copyright or legal issues)
      Restricted
      proprietary hardware drivers etc
      Canonical Partners
      software pkgs by Ubuntu for their partners

    • Sometimes archive.ubunut.com doesn't respond.. change it to us.archive.ubuntu.com 

      sed -i 's/archive\.ubuntu\.com/us\.archive\.ubuntu\.com/' /etc/apt/sources.list

# for EOL, change list
sed -i 's/archive\.ubuntu\.com/old-releases\.ubuntu\.com/g' /etc/apt/sources.list
sed -i 's/security\.ubuntu\.com/old-releases\.ubuntu\.com/g' /etc/apt/sources.list

15.19.1. Package Management

15.19.1.1. Debian only yum
  • yum check-update
    • only checks if any updates are available for installed packages only
  • yum doesn't need to apt-get update like Ubuntu. In fact, yum update updates every currently installed package which is equivalent to Ubuntu apt-get update; apt-get upgrade
  • Try to avoid yum upgrade as it forces the removal of obsolete packages. Same as yum update --obsoletes
  • yum list tree to list package 'tree' info
  • If it returns error, it means the package tree is not installed
15.19.1.2. rpm Red-Hat Package Manager
  • Low level
  • RHEL (Red Hat Enterprise Linux) ,CentOS, Fedora
  • rpm -qa
15.19.1.3. dpkg lowest level Ubuntu, Debian

List installed packages: dpkg -l How to read dpkg output

  • First 3 columns show Desired action, Package status and Error flags
  • Desired action
    • unknown (u), install (i), hold (h), remove (r), purge (p)
  • Package Status
    Not-installed
    n
    Config-files
    c
    Half-installed
    H
    Unpacked
    U
    Half-configured
    F
    Triggers-awaiting
    W
    Triggers-pending
    t
    Installed
    i
  • Error flags
    • <empty> = (none), R = Reinst-required

If a package is installed: dpkg -l wget or dpkg -l | grep ^ii | grep -i wget

More info of an installed package :: dpkg -s openssh-client

See forward/reverse dependencies of a package use apt-cache showpkg javacc

List files owned by a package dpkg -L cron

Find which package owns a file dpkg -S /etc/crontab

15.19.1.4. apt-get apt-cache Ubuntu
  • Since Ubuntu 16.04, apt is introduced and it's a subset of apt-get and apt-cache
  • Update list of available packages /var/lib/apt/lists/
    • apt update
    • apt-get update

  • It might return failed to fetch error. Might be due to linux:dns nameserver is changed to a local ip in a docker container in a docker network. Add a line nameserver 8.8.8.8 to /etc/resolv.conf. This way the /etc/resolv.conf is not persisted. You can also config DNS in Docker daemon 

    RUN echo "nameserver 10.1.1.1" >> /etc/resolv.conf && \
your-other-commands
  • apt-get install openssh-client
  • apt-get -s install openssh-client
  • apt-get upgrade
  • apt-get -V -s upgrade
  • apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
    • Official Debian and Ubuntu docker images automatically run apt-get clean
  • apt list --installed
    • For Ubuntu lower than 14.04, use dpkg -l to list all installed packages on Ubuntu
  • To see if a package is installed
    • apt -qq list xyz
    • apt list -a --installed openssh-client
  • Return info of a package before it's installed
    • apt show php
    • or apt-cache show php
    • or apt list -a --installed php (shows installed if it's installed)
  • apt-cache pkgnames
  • ~apt-cache showpkg php5-xmlrpc=
    • Other packages must be installed before php5-xmlrpc (under Dependencies)
    • Other packages depend on php5-xmlrpc (under Reverse Depends)
    • Different versions of this package you can install (under Provides)
  • apt-cache policy openssh-client
  • https://packages.ubuntu.com/bionic/tmux
15.19.1.5. aptitude Highest Level Debian

It's not installed by default on Ubuntu It removes orphan packages automatically

List installed packages :: aptitude

See if a package is installed :: aptitude search package_name

  • Before each package name, if you see `i` the package is installed
  • v
  • c
  • p
  • There might be a second character which indicates the action to be performed on the package.

15.19.2. Packages

15.19.2.1. openssh-client linux:package:openssh-client

Includes ssh, scp 

apt-get update && apt-get -y install openssh-client
15.19.2.2. supervisord linux:package:supervisord
sudo apt-get install -y supervisor
sudo service supervisor start

# config is at /etc/supervisor/supervisord.conf
# [include]
# files = /etc/supervisor/conf.d/*.conf

# after changing or adding config files
supervisorctl reread
supervisorctl update

# all defined programs in config are run

# go to CLI prompt, C-c to type exit to get out of the supervisorctl tool
supervisorctl

# available commands
supervisor> help

supervisor> stop nodehook
supervisor> start nodehook

Built-in web interface 

[inet_http_server]
port = 9001
username = user # Basic auth username
password = pass # Basic auth password

For example, create a webhooks.conf file to monitor a process 

[program:nodehook]
;; nodehook is a name we give

;; the command to monitor
command=/usr/bin/node /srv/http.js

;; change directory before running the above command
directory=/srv

;; start the process when Supervisord starts (essentially on system boot)
autostart=true

;; the program will be restarted if it exits unexpectedly
autorestart=true

startretries=3

stderr_logfile=/var/log/webhook/nodehook.err.log

stdout_logfile=/var/log/webhook/nodehook.out.log

user=www-data

;; environment variables to pass to the process
environment=SECRET_PASSPHRASE='this is secret',SECRET_TWO='another secret'
15.19.2.3. sendmail 
apt-get install sendmail
# get the hostname of the machine/container
hostname
# say hostname is abc123
cat /etc/hosts

# You should see something like this
127.0.0.1 localhost

# add this line to the end
127.0.0.1 localhost localhost.localdomain abc123

# Run the sendmail config, answer Y to everything, sendmail will be restarted after
sudo sendmailconfig

# restart apache
service apache2 restart

# you may need to run sendmailconfig again to do the config

# check other configs
cat /etc/mail/sendmail.conf
cat /etc/cron.d/sendmail
cat /etc/mail/sendmail.mc

# in case you want to restart sendmail
service sendmail restart
# or
/etc/init.d/sendmail restart

# run this to see which emails are sent/stuck in queue
sendmail -bp
  • Should setup logging in app level, such as php:ini:mail
  • /usr/sbin/sendmail
15.19.2.4. ssmtp
  • ssmtp is a simple alternative to linux:sendmail and it doesn't depend on it
  • sendmail can receive emails
  • sendmail has mail queues
  • ssmtp is good for only one user to send emails
15.19.2.5. postfix linux:postfix
apt-get update && apt install mailutils
# Default is Internet site. Use TAB and ENTER
# System mail name: example.com // Use FQDN (bare domain)
sudo nano /etc/postfix/main.cf

Setup to send only

Change to loopback interface, the virtual network interface that the server uses to communicate internally. 

# inet_interfaces = all
inet_interfaces = loopback-only

Similar to sendmail /etc/hosts setting 

# mydestination = $myhostname, example.com, localhost.com, , localhost
mydestination = $myhostname, localhost.$mydomain, $mydomain

Restart postfix 

/etc/init.d/postfix status
postfix status
postfix start
sudo systemctl restart postfix

# Test
echo "This is the body of the email" | mail -s "This is the subject line" myemail@a.com

# From: sammy@example.com where sammy is Linux username and example.com is the server's hostname

Forward emails sent to root 

sudo nano /etc/aliases

# With the following, system generated emails are sent to the root user.
postmaster: root

# Add
root: myemail@a.com

# restart
sudo newaliases

# Test
echo "This is the body of the email" | mail -s "This is the subject line" root
15.19.2.6. rtorrent
apt-get install rtorrent

Press Enter to enter and you will see load.normal> at the bottom

Paste the link and press enter to load the torrent file, to download it, press up or down arrow and C-s to start the download.

C-q :: quit. Execute twice, it shuts down without sending a stop signal C-s :: start download C-d :: Stop an active download or remove an already stopped download C-k :: stop and close an active download C-r :: hash check a torrent before upload/download begins Left or right arrow :: redirect to previous or next screen Up or down arrow :: to select a torrent file. Selected one will have * in front.

15.19.2.7. Tmux bash:tmux

Man page 

sudo apt install tmux

# version (2.6)
tmux -V

# start tmux to create a new session
tmux
0 is session number, 1 is window number
[0] 1:yourname@your-machine-name:~*
review keyboard shortcuts, press q to exit
C-b ?
(no term)
default bind-key is C-b

Window

Create a new window
C-b c
If you create 3 windows, the bottom looks like
[0] 0:bash 1:bash- 2:bash*
Rename current window
C-b ,
Switch to another window
C-b WindowNumber
Switch to next window
C-b n
Switch to prev window
C-b p
Switch to last window
C-b l
Close current window
C-b &
Show a list of all windows
C-b w
(no term)
* shows which window is active and - shows previous active

Pane

Split current window horizontally into panes
C-b "
Split current window vertically into panes
C-b %
Switch to another pane
C-b o or C-b ArrowKey or C-b ;
Move content in another pane to the current pane
C-b C-o
Resize current pane by increments
C-b C-ArrowKey
Promote a pane to a new window
C-b !
Put a clock into current pane
C-b t and press any key to remove the clock
Close current pane
C-b x

Session

Detach the tmux session and go back to original shell
C-b d
List tmux session
tmux ls
Reattach a tmux session
tmux attach or tmux attach 1
Set tmux session name
C-b $
Swtich to another tmux session
C-b s
Close a session
either close all windows of that session or tmux kill-session -t0

Scroll Mode

Enable navigation mode
C-b [, use arrow keys and q to quit

Copy Mode

Enable mode
C-b PgUp

Command line

Enter command line
C-b :
Quit command line mode
Set option globally
:set-option -g monitor-activity on

Customize ~/.tmux.conf 

# don't allow tmux to automatically rename my already named windows
set-option -g allow-rename off
# Change status bar color
# set -g status-bg cyan
# set bind-key to C-a
set-option -g prefix C-a
unbind C-b
15.19.2.8. OBS Studio - Open Broadcaster Software linux:app:obs-studio
  1. Install

    https://obsproject.com/

    Ubuntu 18.04 

    # Install :: https://github.com/obsproject/obs-studio/wiki/Install-Instructions#linux
# requires xserver-xorg. xserver-xorg-core must be 1.18.4+
apt list -a xserver-xorg
apt list -a xserver-xorg-core

# requires ffmpeg
apt list -a --installed ffmepeg
sudo apt install ffmpeg

# install obs-studio
sudo add-apt-repository ppa:obsproject/obs-studio
sudo apt-get update
sudo apt-get install obs-studio
  2. Usage

    https://www.becomeablogger.com/obs/

    Scene
    a scene contains sources
    (no term)
    Source
    • WebCam
    • Window
      Right click > Preview scaling
      Scale to Window
      Right click > Transform
      Fit to screen
    • Image
    • you can use a text file and modify the file while recording
    (no term)
    Quickly switch between scenes
    Scene1
    Webcam Only
    Scene2
    Webcam + Image
    Scene3
    Browser + Webcam in small window
    Studio mode
    adjust one scene before it goes live
    • click on Scene1 and go to Studio Mode. Now the recording is showing Scene1
    • In Studio Mode, create Scene3 and adjust the browser. Scene3 is not in the recording
    • Then click on Transition insdie Studio Mode to switch to Scene3. Scene3 is live and Scene1 becomes preview

    Settings

    • General
      • Show confirmation dialog when starting streams
      • Show confirmation dialog when stopping streams
    • Output
      Advanced > Recording
      Change save directory
    • Audio
      • Disable Desktop Audio Device and Mic/Auxiliary Audio Device
    • Video
      Base (Canvas) Resolution
      your monitor resolution
      Output (Scaled) Resolution
      your monitor resolution to have perfect quality
  3. Config

    File > Settings

    • Video
      • Change Base (Canvas) Resolution and Output (Scaled) Resolution to 1920x1080 even though my screen is 2560x1440 (16:9)
      • Lanczos
      • 30 (changed from 60)
    • Output
      • Recording
        Recording Format
        flv
        Encoder
        x264
        Rescale Output
        1920x1080
        Custom Muxer Settings
        blank
        Rate Control
        CRF (changed from CBF) for CPU Usage Present ultrafast
        CRF
        15 (should be between 15 and 25)
        Profile
        none
        Tune
        none
        x264 Options
        blank
  4. Audio Filters
    • Compressor
    • Noise Suppression

15.19.3. Ubuntu troubleshooting

15.19.3.1. Unmet dependencies

Run these 

sudo apt-get clean
sudo apt-get update
sudo apt-get -f install
# check if it returns error that about disk full error
# fix disk full error first
# install the package again
sudo apt-get install make

If it still doesn't work or it has error, run these 

sudo dpkg --configure -a
sudo apt-get -f install

If the output is below, it means it failed. 

0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

Next solution is to run 

sudo apt-get -u dist-upgrade

If it shows any held packages, it is best to eliminate them. Packages are held because of dependency conflicts that apt cannot resolve. Try this command to find and repair the conflicts: 

sudo apt-get -o Debug::pkgProblemResolver=yes dist-upgrade

If it cannot fix the conflicts, it will exit with: 0 upgraded, 0 newly installed, 0 to remove and 6 not upgraded.

Delete the held packages one by one, running dist-upgrade each time, until there are no more held packages. Then reinstall any needed packages. Be sure to use the –dry-run option, so that you are fully informed of consequences: 

sudo apt-get remove --dry-run package-name

Since removing the package you are trying to install may not be ideal, you might also try finding a repository that has the packages you need to satisfy the dependencies.

Finally, if all else fails, you can attempt to satisfy the dependencies yourself, either by finding and installing the necessary packages, or by installing them from source and then creating “deb” packages for them.

https://askubuntu.com/questions/140246/how-do-i-resolve-unmet-dependencies-after-adding-a-ppa

15.19.3.2. /boot directory is full linux:disk full

If it's full 100% df -h, all apt commands will not run.

Run this to get installed kernels except the currently running one. 

sudo dpkg --list 'linux-image*'|awk '{ if ($1=="ii") print $2}'|grep -v `uname -r`

# or simply sudo dpkg --list 'linux-image*' for all kernels

Also keep track of the 2 newest versions. uname -r to check the current kernel version. e.g. 

linux-image-4.4.0-31-generic
linux-image-4.4.0-36-generic
linux-image-4.4.0-38-generic
linux-image-4.4.0-42-generic
linux-image-4.4.0-45-generic
linux-image-4.4.0-47-generic
linux-image-4.4.0-51-generic
linux-image-4.4.0-53-generic
linux-image-extra-4.4.0-31-generic
linux-image-extra-4.4.0-36-generic
linux-image-extra-4.4.0-38-generic
linux-image-extra-4.4.0-42-generic
linux-image-extra-4.4.0-45-generic
linux-image-extra-4.4.0-47-generic
linux-image-extra-4.4.0-51-generic
linux-image-extra-4.4.0-53-generic

Current kernel version is 4.4.0-34-generic

Remove other kernels except the current one and the 2 newest ones.

Before delete, check which will be deleted first 

ll /boot/*-4.4.0-{31,36,38,42,45,47}-*
sudo rm -rf /boot/*-4.4.0-{31,36,38,42,45,47}-*

df -h will show the space is reduced. Fix apt-get 

sudo apt-get -f install

If you run into an error that includes a line like "Internal Error: Could not find image (/boot/vmlinuz-3.2.0-56-generic)", then run the command sudo apt-get purge linux-image-3.2.0-56-generic (with your appropriate version).

Now apt-get should work. Finally, sudo apt-get autoremove to clear out the old kernel image packages that have been orphaned by the manual boot clean.

Run sudo apt-get update and maybe sudo apt-get upgrade after.

https://askubuntu.com/a/430944/481813

15.20. curl, wget

15.20.1. curl

15.20.1.1. Basic
apt update && apt install -y curl

#!/bin/bash
dlDir=/home/lili/download/
dlFileName= ${dlDir}123.xml
curl "http://abc.com/123.xml" -L -o ${dlFileName}
15.20.1.2. Return HTTP response header only

curl -m 120 -IL "http://url.com" >> /path/to/log.log e.g Drupal Cron URL response status 

#!/bin/bash
curl -sL -o /dev/null -w "%{http_code}\n" "http://drupalcronurl" >> /path/to/your/cronlog
15.20.1.3. Download with HTTPS or HTTP authentication
Popup form or server authentication
curl https://a.com/b.aspx -u user:password -L -o /path/to/xml.xml
15.20.1.4. cURL download only after or before a date

Works for ftp and Http 

fileLocalTime=$(date -r "$localfile")
curl $remote_file_path -u $user:$password -o $localfile -R -z "${fileLocalTime}"

# default is "after"
# add a dash before date string will result in "before".
fileLocalTimeNew=$(date -r "$localfile")

if [ "$fileLocalTime" == "$fileLocalTimeNew" ]; then
 echo "the same"
else
 echo $fileLocalTime
 echo $fileLocalTimeNew
fi
15.20.1.5. HTTP POST 
# default POST is application/x-www-form-urlencoded
curl -X POST -d "test=that&test2=that2" http://..

# explict -H 'Content-Type: application/x-www-form-urlencoded'

# with a data file
curl -X POST -d "@data.txt" http://..

# json
# keep value to be a single line
# I haven't figured out how to escape newline for json yet..
to='a@a.ca'
from='b@a.ca'
json_fmt='{"personalizations": [{"to": [{"email": "%s"}]}],"from": {"email": "%s"},"subject": "%s","content": [{"type": "text/plain", "value": "%s"}]}'
data_json=$(printf "$json_fmt" "$to" "$from" "$subject" "$content")

curl -X POST -H 'Content-Type: application/json' -d "{\"test\": \"that\"}" http://..
curl -X POST -H 'Content-Type: application/json' -d '{"test":"that"}' http://..
curl -X POST -H 'Content-Type: application/json' -d "$data_json" http://..

# with a data file
curl -X POST -d "@data.json" http://..
15.20.1.6. Use different IP address from DNS

a.ca by DNS points to 1.2.3.4 IP, change it to a different IP 2.2.3.4 temporarily. 

curl -L -v -I --resolve 'a.ca:80:2.2.3.4' http://a.ca

# https://a.ca still points to 1.2.3.4 as port 80 does not match the request a.ca@443
curl -L -v -I --resolve 'a.ca:80:2.2.3.4' https://a.ca

# Use this instead
curl -L -v -I --resolve 'a.ca:443:2.2.3.4' https://a.ca

# sub domain
curl -L -v -I --resolve 'www.a.ca:443:2.2.3.4' https://www.a.ca

# --resolve <host:port:address> force resolve of HOST:PORT to ADDRESS
15.20.1.7. Fetch web pages and save as HTML files

Uses bash:array 

#!/bin/bash
set -o history -o histexpand

declare -A fileMaps=(
[https://a.com/]='pages/index.html' \
[https://a.com/page1]='pages/page1.html' \
[https://a.com/page-2]='pages/page-2.html' \
)

declare -a curlArgs=('-L' '-H' "Connection: keep-alive" '-H' "Pragma: no-cache" '-H' "Cache-Control: no-cache" '-H' "Upgrade-Insecure-Requests: 1" '-H' "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" '-H' "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8" '-H' "Accept-Encoding: gzip, deflate, br" '-H' "Accept-Language: en-US,en;q=0.9,zh-CN;q=0.8,zh-TW;q=0.7,zh;q=0.6")

for K in "${!fileMaps[@]}"; do
 curlArgsTemp=("$K" "${curlArgs[@]}")
 curlArgsTemp=( "${curlArgsTemp[@]}" "-o" "${fileMaps[$K]}" )
 curl "${curlArgsTemp[@]}"
done

# echo !!
15.20.1.8. Fetch as Googlebot curl:ua:googlebot

Googlebot's user agent: https://support.google.com/webmasters/answer/1061943 

curl -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -v https://a.ca
15.20.1.9. Options
-L Follow through all redirects
-o redirect STDOUT to a file
-O save file using same file name in current folder
-I get header response only
-m timeout in seconds
-H request header. -H "Accept-Charset: ISO-8859-1,utf-8;q=0.7"
–limit-rate limit download speed: –limit-rate=200k. use k, m or g
-s suppress STDOUT. -svo /dev/null show req/resp headers only
-w rewrite STDOUT
-R Keep remote file timestamp. –remote-time
-z cURL download only after or before a date
-d send POST data
-A --user-agent
-v verbose
-b, –cookie -b "name1=v1; name2=v2"

15.20.2. wget bash:wget

Compared to curl, wget's advantages are:

  • recursive
  • continue downloading after broken transfer
  • Specify cookies in get request
  • Has redirect-following
  • Capture response such as time stamping from the remote resource.

Disadvantages are only HTTP/HTTPS/FTP, Basic auth over HTTP proxy, no SOCKS support

wget [option]… [URL]…

Run a wp cron: 

# /dev/null
wget -O /dev/stdout -o /dev/stdout "https://mysite.ca/wp-cron.php?import_key=123&import_id=8&action=processing"

http://[username:password@]host[:port]/directory/file
ftp://[username:password@]host[:port]/directory/file

Download all subfolders and files of an FTP folder 

cd ~
wget --user=hello --password='mypassword' -cr ftp://server/path/to/test

# Default depth is 5, change it
wget -r -l 2 ftp://... 
# for infinite depth :: -l inf

# path/to/test is 3 depths, by default, 2 more levels can be downloaded

# ~/server/path/to/test/* will be downloaded

# -nH ~/path/to/test will be downloaded
# -nH --cut-dirs=1 ~/to/test will be downloaded
# -nH --cut-dirs=2 ~/test will be downloaded
# --cut-dirs=1 ~/server/to/test will be downloaded

# --ftp-user or --http-user can also be used. There's no difference

# Ignore folders (not file)
wget -cr -X path/to/folder/ignore/ ftp://server/path/to/folder/

# folder path/to/folder/ignore/ will not be downloaded

# wildcards can also be used
wget -cr -X path/to/folder/ignore*/ ftp://server/path/to/folder/

# ignore multiple folders
-X path/to/folder/ignore1,path/to/folder/ignore2

# Only include certain folders -I, exactly the same as -X

# Only download files -A
-A books*,zelazny*196[0-9],gif,jpg

gif and jpg are normal letters, they match the ending part of a file (not exactly file extension)

# Ignore certain files -R, exactly the same as -A

# -nH :: --no-host-directories :: use it with --cut-dirs. e.g. download http://a.com/ by default creates a.com/ directory

# -P :: --directory-prefix :: Default is . (the current directory)

# -N to retrieve timestamps so if local files are newer, skip download.

# -c :: enable continue download

# --limit-rate=20k :: limit download speed to 20kb/s, m or g can also be used

# -b :: put command run in background

# -O path/to/local/file :: output to a local file. Change downloaded filename

# Logging
# -v :: default
# -q :: complete quiet
# -nv :: only error and basic info are printed

Manual

15.21. Bash File, Input and Output

15.21.1. Version echo $BASH_VERSION

15.21.2. Shebang

  • #!/usr/bin/bash
    • Absolute path to a shell
  • #!/usr/bin/env php
    • Environment variables are loaded and use the default shell to run command php
      • Environment variable SHELL defines the default shell and it's loaded in the files listed in environment variable PATH

15.21.3. Last directory ~-

cd ~/abc
cd xyz
echo ~-

15.21.4. Directory of the curren script file

# Normal usage when there is no symlink
# Result has no trailing `/`
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

# When symlinks are involved
SOURCE="${BASH_SOURCE[0]}"
while [ -h "$SOURCE" ]; do # resolve $SOURCE until the file is no longer a symlink
  DIR="$( cd -P "$( dirname "$SOURCE" )" >/dev/null 2>&1 && pwd )"
  SOURCE="$(readlink "$SOURCE")"
  [[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
done
DIR="$( cd -P "$( dirname "$SOURCE" )" >/dev/null 2>&1 && pwd )"

15.21.5. range bash:range

echo {1..100}
# 100 numbers from 1 to 100

touch file_{1..100}

# add padding (one zero in front is enough)
echo {01..100}
# 01 02 ... 100

# range with increment other than 1
echo {1..10..2}
# 1 3 5 7 9

echo {1..10..3}
# 1 4 7 10

echo {A..Z}
# A to Z

echo {A..z}
# A to z

echo {a..Z}
# not a to Z!
# a ` _ ^ ] [ Z

echo {w..d..2}
# w u s q o m k i g e

# customize range
touch {apple,banna,cherry,durian}_{01..100}{w..d}

15.21.6. File Descriptors, Redirection Operator, Piping

  • Use linux:tee to redirect output to a file and display output at the same time
  • File descriptors
    • First 3 are reserved
    • STDIN Standard input
    • STDOUT Standard output e.g. echo
    • STDERR Standard error

    • Up to 9 open file descriptors can be used at a time 

      # Open a custom file descriptor to a file
exec 3>customLog

# redirect output to file descriptor 3.

# `&` on the right is just required for syntax
# `&` on the left means both 1 and 2 file descriptors
echo "This goes to file descriptor 3" >&3

# Open a custom file descriptor and redirects to STDOUT
exec 4>&1

# If you want to change the reserved file descriptors. Always backup and change back
exec 5>&1
exec 6>&2

# Backup STDIN is a little different
exec 7<&0

  • > and < 

    # `&` on the right is just required for syntax
# `&` on the left means both 1 and 2 file descriptors

# Redirect stdout, creates if not exist or overwrites a file
ls -al goodfile > stdoutlogfile
# same as 2>, 3> etc.

# Redirect stdout, creates if not exist and appends to a file
ls -al goodfile >> stdoutlogfile
# same as &>>, 2>>, 3>> etc.

# Redirect stderr only
ls -al badfile 2> errorlogfile

# Redirect stdout and stderr to different files
ls -al goodfile badfile 2> errorlogfile 1> normalOutput

# Redirect stdout and stderr to the same file
ls -al goodfile badfile &> errorAndOutput

# Redirect stdout and stderr to the same file by appending
ls -al goodfile badfile &>> errorAndOutput

# suppress stdout and stderr
ls -al goodfile badfile &> /dev/null

# Usage of >, &>, 2>>, etc
# On the left, it's an output not a file

# Usage of <
# On the right, it's a file not an output
sort < fruit.txt
wc < fruit.txt
# clear a file's content, perfect when sudo is not needed
< clearThisFile.txt

  • Piping 

    # on the left, output
# on the right, command that takes input
echo "Hello" | wc
15.21.6.1. tee

  • Basics 

    # Redirect stdout of a commannd to a file and display stdout
date | tee testfile

# Redirect stdout and sterr of a command to a file and display as stdout
date 2>&1 | tee testfile

# tee can be used to bypass pagination (press q to quit)
man ls | tee

# Redirect stdout to a file and pass stdout to another command as stdin
date | tee testfile | less

# Capture stdout of a command to a variable
output=$(date)

# Capture stdout and stderr of a command to a variable
output=$(command_which_has_stdout_and_stderr 2>&1)

# Redirect stdout of a command to one or multiple files and save stdout to a variable
# a file must be behind a tee command
# By default tee overwrites the file. Use -a to append to the end
$output = $(date | tee -a testfile1 testfile2)

# Redirect stdout to a variable and display to terminal in real time (not to stdout!)
# Works on Ubuntu and RedHat
$output = $(date | tee /dev/tty)

# Redirect one file descriptor (M, default 1) to another file descriptor (N) M>&N
# Redirect a manually made error 
echo "This is an error" >&2

# Change a file descriptor to a file
exec 2>errorLogFile
echo "This is STDOUT and STDOUT is not changed to a file yet"
exec 1>stdOutFile
echo "This doesn't display as STDOUT is redirected to a file"
echo "This is an error and in errorLogFile" >&2
  • linux:cp:multiple destinations

15.21.7. declare bash:declare

# integer
declare -i d=123

# read only, var can't be changed
declare -r e=456

o=hello

# uppercase -u, lowercase -l
declare -u upper=$o

15.21.8. Array bash:array

# define empty array
myarray=()

# define indexed array with values
myarray=( one two three four five )
declare -a curlArgs=('-H' "keyheader: value" '-H' "2ndkeyheader: 2ndvalue")

# run `help declare`
# declare -a for indexed array
# declare -A for associative array

# define array with strings
myarray=( 'Hello World' 'Another item' )

# Get at index 0
echo ${myarray[0]}

# print array as a whole delimited by space
echo ${myarray[@]}

# print array as a whole delimited by newline
printf "%s\n" "${myarray[@]}"

# Convert an array to string variable delimited by newline
printf -v stringvar "%s\n" "${myarray[@]}"

# Remove the final newline
stringvar=${stringvar%?}

# prepend
arr=("new_element1" "new_element2" "..." "new_elementN" "${arr[@]}")

# Append to array
arr=( "${arr[@]}" "new_element" )
myarray+=('foo')
myarray+=("$line")

# Append only if it's new
containsElement() {
 local e
 for e in "${@:2}"; do [[ "$e" == "$1" ]] && return 0; done
 return 1 
}
if ! containElement "$line" "${myarray[@]}"; then
 myarray+=("$line")
fi

# Append to a specific index (e.g. at arr[2])
arr=( "${arr[@]:0:2}" "new_element" "${arr[@]:2}" )

# Remove an element at a specific index (e.g. arr[2])
arr=( "${arr[@]:0:2}" "${arr[@]:3}" )

# Length
echo Number of elements in array: ${#myarray[@]}

# Loop array
for i in "${myarray[@]}"; do
  echo "$i"
done
15.21.8.1. Associative array
declare -A mymap
mymap[foo]=bar
echo ${mymap[foo]}

declare -A MYMAP=( [foo]=bar [baz]=quux [corge]=grault )

k=baz
mymap[$k]=quux
${mymap[$k]} # is the same as ${mymap[baz]}

Key with spaces can be double and single quoted but they are the same as not quoting them
mymap[foo a]="bar b"
mymap["foo a"]="bar b"
mymap['foo a']="bar b"

# Testing whether a value is missing from an associative array
if [ ${MYMAP[foo]+_} ]; then echo "Found"; else echo "Not found"; fi
# Found

if [ ${MYMAP[missing]+_} ]; then echo "Found"; else echo "Not found"; fi
# Not found

declare -A MYMAP=( [foo a]=bar [baz b]=quux )
echo "${!MYMAP[@]}"  # Print all keys - quoted, but quotes removed by echo
# foo a baz b

# Loop through all keys in an associative array
for K in "${!MYMAP[@]}"; do echo $K; done
# foo a
# baz b

# Loop through keys and values in an associative array
for K in "${!MYMAP[@]}"; do echo $K --- ${MYMAP[$K]}; done
# foo a --- bar
#baz b --- quux

# Redeclare, clear the whole array
declare -A MYMAP
MYMAP[foo]=bar
echo ${MYMAP[foo]}
# bar

declare -A MYMAP    # Re-declaring DOES NOT clear an associative array
echo ${MYMAP[foo]}
# bar

unset MYMAP         # You need to unset and re-declare to get a cleared associative array
declare -A MYMAP
echo ${MYMAP[foo]}

# Unset a key
MYMAP[foo]=bar
echo ${MYMAP[foo]}
# bar

unset ${MYMAP[foo]} # WRONG
echo ${MYMAP[foo]}
# bar

unset MYMAP[foo]    # To delete from an associative array, use "unset" with similar syntax to assigning
            # BUT see next section if key contains spaces
echo ${MYMAP[foo]}

MYMAP[baz]=quux
echo ${MYMAP[baz]}
# quux

K=baz
unset MYMAP[$K]       # Can unset using a variable for the key too
          # BUT see next section if variable may contain spaces - always better to quote
echo ${MYMAP[baz]}

# Unset a key that has spaces
unset MYMAP[foo Z]     # WRONG
unset MYMAP["foo Z"]    # You must quote keys containing spaces when you unset in an associative array
echo ${MYMAP[foo Z]}

# Length, same as indexed array
declare -A MYMAP=( [foo a]=bar [baz b]=quux )
echo ${#MYMAP[@]}  # Number of keys in an associative array

15.21.9. echo, print

# without quotes, need to escape special characters
echo $o, world \(planet\)!

# single quotes will not intepret variables
echo '$o, world (planet)!' # $o, world (planet)!

# enable backslash escape
echo -e "Hello\nWorld"; 
# echo newline -e to use \n etc
# always wrap variable in double quotes to preserve newline character

15.21.10. printf bash:printf

# e.g. Print multilines
printf '%s\n%s\n' "$line1" "$line2"

15.21.11. Here document, here string bash:heredoc

  • The following is equivalent to some-interactive-script < command-file 

    some-interactive-script <<EOF
command #1
$varName
EOF
  • suppress leading tabs in the document body
  • don't expand variables/command
    can be combined with -
    <<-'EOF'

  • Store as a variable 

    read -r -d '' VAR <<EOM
This is line 1.
This is line 2.
Line 3.
EOM

echo "$VAR"
  • http://tldp.org/LDP/abs/html/here-docs.html
  • Here string

    • some-interactive-script <<<$myCommand or some-interactive-script <<<"$myCommand"

      • wc -w <<< "This is a test."
      • grep "nor" <<<$var >/dev/null && echo "Found" || echo "Not found" equivalent to

      echo $var | grep -q "nor" && echo "Found" || echo "Not found"

15.21.12. Read command output line by line, Prompt

15.21.12.1. Read command output line by line
multiple_commands_output=""
output=$(drush command output contains multiple lines);

# more examples
output=$(docker stats --no-stream --format "{{.Name}}: {{.CPUPerc}}")

# a file
inputfile="/path/to/txt/file"

count=1
while IFS=$'\n' read -r line; do
  echo "Line number $count: $line"
  count=$[ $count + 1 ]

  # refer to bash:heredoc
done <<<"$output"

# for a file
# done <"$inputfile"

multiple_commands_output="$multiple_commands_output""$output"$'\n'

# You don't have to change back IFS as it only affects the while loop
# newline character is \n
# read -r :: to ignore all backslashes
15.21.12.2. Prompt
# `-n 1` accept one character
# `-p` outputs a prompt
# `-r` accepts backslashes literally otherwise it would see the backslash as an escape and wait for a second character
# `$REPLY` is the default global variable the command uses to store the answer input
# To specify a variable: read -p "..." -r my_var
read -p "Are you sure? [Y/n]" -n 1 -r
echo    # (optional) move to a new line

# `=~` to match a regex 
if [[ ! $REPLY =~ ^[Yy]$ ]]
then
    exit 1
fi

15.21.13. Date format

date +%F
YYYY-MM-DD
date +%M
minute 00..59
date +%H
00..23
date +%S
second 00..60
date +%T
HH:MM:SS
date +%Z
time zone e.g. EDT
date +%FT%T%Z
2017-10-13T16:39:02UTC
date +%FT%H%M%S
2017-10-13T163902 use this as it is filename safe
date +%FT%H%M%S%Z
2017-10-13T163902EDT use this as it is filename safe
mydate=$(date +"%m-%d=%Y")
bash variable

15.21.14. Last executed command

#!/bin/bash
set -o history -o histexpand
var1='hello'
echo $var1
echo !!

15.21.15. Passing arguments to command bash:pass arguments

It is ok to pass variables as argument value which is preceded by the argument name 

ua='User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'
curl https://a.ca/ -L \
  -H "$ua"

Use bash:array to pass argument name and value 

# create a non-associative array
declare -a curlArgs=('-H' "keyheader: value" '-H' "2ndkeyheader: 2ndvalue")
curl "${curlArgs[@]}"

# args inside args

declare -a historyDirs=(":(icase)wp-content/themes" ":(icase)wp-content/plugins")
declare -a gitlogArgs=("--author=$j" '--since=last 1 month' '--pretty=tformat:' '--numstat' '--' "${historyDirs[@]}")
git log "${gitlogArgs}"

15.21.16. String Operators

# A string is delimited by :
# Get the last element
foo=1:2:3:4:5
echo ${foo##*:}

# '##' means greedy front trim
# * matches any character
# until the last ':'
# also work for space as delimiter
# ${foo##* }

# Concatenate string variables
z=""
a="Hello\n"
b="World\n"
c="!"

z="$z""$a"
z="$z""$b"

# insert a newline
z="$z""$c"$'\n'

15.21.17. String comparison

# Start with 'node'
if [[ $a == node* ]]; then

# Start with a string 'Hello World'
if [[ $a == "Hello World"* ]]; then

15.21.18. Numeric comparison

Not floating-point values

Equal $n1 -eq $n2
greater or equal $n1 -ge $n2
  $n1 -gt $n2
  $n1 -le $n2
  $n1 -lt $n2
not equal $n1 -ne $n2
   

15.21.19. Bash: shell variable with default value

  • ./yourcustom.sh var1 var2 

    this_var1=$1
this_var2=${10}

# maximum 11 variables
this_var3=${3:-"_"}
# if var3 is not provided, then default is string _

15.21.20. if bash:if

  • Exit code zero then conditions is true
  • ; & && ||
if [ conditions ]; then
 # commands
elif [ conditions ]; then
 # more commands
elif [ conditions ]; then
 # more commands
else
 # more commands
fi

if [ -z "$(ls -lA)" ]; then
  echo "no files found"
else
  echo "There're files"
fi

# if [ condition ] :: all POSIX shells
# if [[ condition ]] :: new upgrad e.g. whether a string matches a regular expression. Supported by ksh, bash and zsh
# if ((condition)) :: Supported by ksh extension, b ash and zsh. Return zero if the result of the arithmetic calculation is nonzero
# if (command) :: Run a command in a subsehll
# if command :: Run a command
! EXPRESSION
The EXPRESSION is false
-n STRING
The length of STRING is greater than zero
-z STRING
The lengh of STRING is zero (ie it is empty)
STRING1 = STRING2
STRING1 is equal to STRING2
STRING1 != STRING2
STRING1 is not equal to STRING2
INTEGER1 -eq INTEGER2
INTEGER1 is numerically equal to INTEGER2
INTEGER1 -gt INTEGER2
INTEGER1 is numerically greater than INTEGER2
INTEGER1 -lt INTEGER2
INTEGER1 is numerically less than INTEGER2
-d FILE
FILE exists and is a directory
-e "$myfile"
FILE exists
-f "$myfile"
File is a regular file (not a dir or device file)
-r FILE
FILE exists and the read permission is granted
-s "$myfile"
FILE exists and it's size is greater than zero (ie. it is not empty)
-w FILE
FILE exists and the write permission is granted
-x FILE
FILE exists and the execute permission is granted

15.21.21. Function bash:function

function_name () {
}

# or

function function_name {
}

# Pass arguments

print_something () {
  local var1='local var1'
  echo Hello $1
  echo $var1
  # local var1
  # global var1 is not changed
  echo $var2
  # global var2
  return $1
}

var1='global var1'
var2='global var2'
print_something Mars

output=$( print_something "Mars" )
output=$( print_something "$1" )

15.21.22. Special Parameters

  • man bash
  • expands to "$1" "$2" "$3" ...
  • expands to the name of the shell script file

  • $* 

    for arg in $*
do
    echo "Command line arg: $arg"
done

# ./run.sh One Two Three
# Command line arg: One
# Command line arg: Two
# Command line arg: Three

for arg in "$*"
do
    echo "Command line arg: $arg"
done

# ./run.sh One Two Three
# Command line arg: One Two Three
  • number of arguments
  • exit status of last run command
  • process ID of the current shell
  • the last command
  • last argument of the last command

15.22. Cron

15.22.1. Install

apt install cron
sudo systemctl enable cron
sudo systemctl start cron
sudo systemctl status cron

15.22.2. Explain https://crontab.guru/

15.22.3. Cron table

  • Each user can have his own cron table. Use crontab to manage the current user's cron jobs
  • crontab -l
  • crontab -u username -l
  • All users' crontabs
    Each user
    sudo ls -al /var/spool/cron/crontabs or sudo ls -al /var/spool/cron/
  • cat /etc/crontab
  • Cron jobs of packages usually saved under /etc/cron.d/* use the same syntax as /etc/crontab/
  • 4 cron directories. Copy script files to one of them and and they will be run as root
    • ls -al /etc/cron.*
    • /etc/cron.daily, /etc/cron.hourly, /etc/cron.montly, /etc/cron.weekly
  • crontab -l > ~/cron.back
  • crontab -e

echo $PATH may have to be copied to either the script file or crontab because env in cron is different from env the shell that you run the script in. 

# Setup a Linux cron to run a Drupal Cron URL every 5 minutes
*/5 * * * * /path/to/rundrupalcron.sh

15.22.4. User crontab syntax and system-wide syntax

User Crontab Syntax min hour dayofmonth month dayofweek command

  • /var/spool/cron/crontabs/yourusername or /var/spool/cron/*

System Crontab Syntax min hour dayofmonth month dayofweek user command

  • /etc/crontab

range :: 1-5 wildcard :: *

dayofweek :: can be (mon, tue, wed, thu, fri, sat, sun) or (1, 2, 3, 4, 5, 6, 0) 

# Last day of month 
00 12 * * * if [ `date +%d -d tomorrow` = 01 ] ; then ; command

# Every 15 minutes every Wednesday
*/15 * * * 3 echo "do something

# Every hour except 6am 
00 0-5,7-23 * * * echo "hi"

Special strings to replace with all 5 time-and-date fields

@reboot Run once, at startup.
@yearly Run once a year, "0 0 1 1 *".
@annually (same as @yearly)
@monthly Run once a month, "0 0 1 * *".
@weekly Run once a week, "0 0 * * 0".
@daily Run once a day, "0 0 * * *".
@midnight (same as @daily)
@hourly Run once an hour, "0 * * * *".
   

15.22.5. Sample

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
0 2 * * * /home/li/cron/a.sh
10 2 * * * /home/li/cron/b.sh
*/20 0-2,6-23 * * * /home/li/cron/c.sh
10 3 * * * /home/li/cron/d.sh
10 19 * * 0 /home/li/cron/e.sh

15.22.6. Non cron way

Every 5 seconds 

#!/bin/bash
while true; do
  echo "hi"
  sleep 5
done

15.23. InCron - inotify cron 

sudo apt-get install incron

# /etc/, the following are created
# - incron.allow
# - incron.conf
# - incrond.d/
# - incron.deny
# one user per line e.g. root
nano /etc/incron.allow

# https://linux.die.net/man/8/incrond

systemctl start incron.service
# or
/etc/init.d/incrond start

incrontab -l
incrontab -e

# <path/to/folder/or/file> <mask> <command>

# Mask: https://linux.die.net/man/5/incrontab

# IN_CLOSE_WRITE :: file opened for writing was closed.
# IN_MODIFY :: file was modified. Before file open for writing is closed, there may be more than 1 time the file/folder is modified

# Wildcards
# $$ dollar sign
# $@ watched filesystem path (see above)
# $# event-related file name
# $% event flags (textually)
# $& event flags (numerically)

# example
/path/to/file IN_CLOSE_WRITE /usr/bin/php /home/vmail/ma-notify.php $@ $# $%

15.24. upx linux:upx

Executable file packer (compresser) 

apt-get install upx-ucl

# Most compressed level. Result file go.upx
upx --brute go

15.25. Commands

15.25.1. xclip bash:xclip

sudo apt install xclip

# select/copy
xclip -sel clip < ~/.ssh/id_rsa.pub

15.25.2. screen bash:screen

It creates a screen session

  • In a screen session, multiple shell windows (virtual terminals) can be open from a single SSH session.
    • Processes running in Screen will continue to run when their window is not visible even if the SSH session is terminated
    • If network connection fails in one screen session, screen will automatically detach the session.
  • https://www.howtogeek.com/662422/how-to-use-linuxs-screen-command/

  • Installation 

    which screen
sudo apt update
sudo apt install screen
  • C-a ?

  • Commands about screen session 

    # open a screen session with a new window with a new shell
screen

# To create a screen session with a name
screen -S my_session_name

# Detach the current screen session
C-a d

# List running screen sessions
screen -ls

# Restore a detached screen session using session ID
screen -r 10835

# simply quit or exit all windows in the attached session will close the session
quit
exit

# To close a session
# - -X: execute command
# - -S: session PID
screen -XS 10835 quit

  • Commands about windows in a screen session 

    # Create a window with shell inside a screen session
# ~C-a c~

# Switch between windows
# - next window ~C-a n~
# - previous window ~C-a p~
# - Go to a specific window ~C-a 0~ ~C-a 1~

  • Commands about regions in a screen session 

    # To split a window into 2 regions: the first region is the current window, the second region is empty
# - Split the window horizontally: ~C-a S-s~
# - Split the window vertically: ~C-a |~

# Move cursor to the next region (the empty region)
# ~C-a Tab~

# Display a window in a new region
# - choose to open the first window: ~C-a 0~
# - Or create a new window: ~C-a c~

# Region navigation
# - Go to next region: ~C-a~
# - Close all regions except the current one: ~C-a q~
# - Close the current region: ~C-a x~
  • Get alert
    • To get an alert when there's output registered on a specific screen, on that screen run Ctrl-a M and then switch to other screen
    • Or get an alert when there's no output C-a _
  • C-a x
  • C-a k or exit for each screen window.
  • screen -ls
  • C-a " it's double quote S-'
  • C-a A

Run a background process after SSH signout 

screen
nohup ./main &
# C-a d
exit

15.25.3. Run Sequentially

lspci; lsusb If the first command not successful, 2nd command will not run: lspci && lspci

15.25.4. ls

List files or directories by name

  • ll
    Ubuntu server default
    ls -alF
  • Options
    sort by time last modified
    -t
    sort by file size
    -S
    reverse sort
    r
    in detail
    l
    In MB
    -l --block-size=M
    Append indicator to entries
    -F, –classify
    @
    symbolic link (or that the file has extended attributes)
    *
    executable
    =
    socket
    |
    named pipe
    >
    door
    /
    directory
    Output immediately
    -1
    do not sort; list entries in directory order
    -U
    do not sort, enable -aU, disable -ls –color
    -f
  • Results of ls -l
    • -rwxr-xr-x 1 10490 floppy 17242 May 8 2013 acroread
    • First - represents a regular file. It could be
      d
      directory
      c
      character device
      l
      symlink
      p
      named pipe
      s
      socket
      b
      block device
      D
      door
      -
      regular file
  • Large directory
    ll -U -1
    do not sort, print immediately
    ll -1 -f
    do not sort, print immediately without color
ls -l my_scr?pt
ls -l my*
ls -l my_s*t
ls -l my_scr[ai]t
ls -l f[a-i]ll
ls -l f[!a]ll
15.25.4.1. colors
  1. Bash shell
    eval $(echo "no:global default;fi:normal file;di:directory;ln:symbolic link;pi:named pipe;so:socket;do:door;bd:block device;cd:character device;or:orphan symlink;mi:missing file;su:set uid;sg:set gid;tw:sticky other writable;ow:other writable;st:sticky;ex:executable;"|sed -e 's/:/="/g; s/\;/"\n/g')           
{      
  IFS=:     
  for i in $LS_COLORS     
  do        
    echo -e "\e[${i#*=}m$( x=${i%=*}; [ "${!x}" ] && echo "${!x}" || echo "$x" )\e[m" 
  done       
}

15.25.5. less 

less afile.txt

# show line numbers at the footer
less -M afile.txt

# show line numbers on the left
less -N afile.txt
Go back and up one page
f b
Go to start and end of the file
g S-g
exit
q

15.25.6. head, tail

Last 5 lines
tail -n5 /path/to/file
Keep tailing
tail -f /var/log/messages
Used in chain
ls -t | tail -n3
exit
C-c

15.25.7. man

Search processes in command names and description
man -k processes
Search in command names and title
man -f ls
Save manual to a txt file
man ls > ls.txt

15.25.8. cd - Navigate to previous directory (back)

15.25.9. Word Count wc

  • wc filename
    Output
    13 14 123 filename
    • 13 lines, 14 words, 123 characters
  • Line Count e.g. ps -ef | wc -l
  • Word Count
  • Byte Count
  • Character Count

15.25.10. sort

  • sort filename
    • output sorted lines. File is not changed
  • Options
    • -f, --ignore-case
    • -r, --reverse
    • -u, --unique

15.25.11. uniq

  • uniq filename
    • Different from linux:sort, uniq dedup consecutive lines
  • uniq -d filename
    • Find lines that appear at least twice in consecutive order
  • uniq -u filename
    • Filter out lines that repeated consecutively

15.25.12. Replace String, Stream Editor sed

  • Replace all (g) Nick with John case-insensitive
    • cat report.txt | sed 's/Nick/John/gi' > report_new.txt
  • sed SCRIPT INPUTFILE...
    • or sed OPTIONS... [SCRIPT] [INPUTFILE...]
  • SCRIPT
    s command
    substitution
    • 's/{regexp}/{replacement}/{flags}'
    • To literally match string, wrap each letter in [ and ] and for ^ do [\^]
    e.g. literal string '&#x1f;'
    sed 's/[&][#][x][1][f][;]//gi' -i c.xml
    literal string '&#^x1f;'
    sed 's/[&][#][\^][x][1][f][;]//gi' -i c.xml
    (no term)
    Special characters to escape for {regexp}
    Single quote
    \'
    $
    \$
    .
    \.
    *
    \*
    [
    \[
    \
    \\
    ]
    \]
    ^
    \^
    `
    \`
    (no term)
    {replacement}
    (no term)
    e.g. echo 'daytime' | sed -E 's/(...)time/\1light/', result is ~xxxlight
    • Up to \9
    (no term)
    Use & to subsitute the whole matched portion of the pattern
    (no term)
    Special characters to escape for {replacement}
    Single quote
    \'
    (no term)
    & and \ need to be escaped, as do the delimiter (usually /) and newlines \n
    (no term)
    Refer to regexp and replacement escape
    (no term)
    Instead of / as delimiter, use | for handling string with / such as http://
    • Always use \ to escape no matter what delimiter is
    (no term)
    For inside bracket expression (list of characters), to literally include:
    ^
    put it not at the beginning [ab^c]. If it's the only one, use [\^]
    -
    put it either at the start or end [-abc] or [abc-] not [a-bc]. If it's the only one, use [\-]
    ]
    put it at the start []abc] or [^]abc] but not [abc]] nor [abc\]]. If it's the only one, [^]]
    Notice
    escape \ is crucial because \number e.g. \1 is subsitution, \letter e.g. \n also has special meaning
    Notice
    use double quotes for interpolation sed -e "s/$BRE/$REPL/"
  • OPTIONS
    -E
    extended regex
    -e SCRIPT
    run several inline SCRIPT's described above
    -f SCRIPT-FILE
    Run commands in SCRIPT-FILE's
    -i
    files are edited in-place. Original files are overwritten, no backup files are created
    Mac

    cavat 

    sed -i "s/ENCRYPTION='Y'//g" ../../../mysql-ma-bk/backup.sql
# sed: 1: "../../../mysql-ma-bk/ba ...": invalid command code .

# Should be
sed -i '' -e "s/ENCRYPTION='Y'//g" ../../../mysql-ma-bk/backup.sql
    (no term)
    For deleting only
    • sed "s/deletethis//g" -i input.txt
    • sed 's/&#x1f;//gi' -i c.xml
    -i.backup or -i .backup (only Mac)
    copy original to *.backup and change original files
    • For multiple files
      No backup
      sed 's/Nick/John/gi' *.txt
      create backup files
      sed 's/Nick/John/gi' -i.backup *.txt
      (no term)
      ::
    (no term)

    -n, --quiet, --silent 

    # Assume the file has
# The quick brown fox jumps over the lazy dog.

# Notice :: there're 2 substitution commands here
sed -e 's/fox/coati/;s/dog/dingo/' myfile

# prints output after all commands
# The quick brown coati jumps over the lazy dingo.

# suppress output
sed -n -e 's/fox/coati/;s/dog/dingo/' myfile

# suppress output but print result after the first command
sed -n -e 's/fox/coati/p;s/dog/dingo/' myfile
# The quick brown coati jumps over the lazy dog.

# p can be a standalone command. These 2 commands are equivalent
sed -n -e 's/fox/coati/p;s/dog/dingo/' myfile
sed -n -e 's/fox/coati/;p;s/dog/dingo/' myfile

# These 2 commands print out the final output
sed -n -e 's/fox/coati/;s/dog/dingo/;p' myfile
sed -n -e 's/fox/coati/;s/dog/dingo/p' myfile
  • man sed
  • Refer usage in mysql:unknown collation

15.25.13. xargs

  • Build and execute commands from standard input. It converts input from standard input into arguments to a command
echo 'lorem.txt' | xargs wc

# output whole command before run (-t)
echo 'lorem.txt' | xargs -t wc

# loop
# `-n 1` means use at most 1 argument per command line

echo 'a.txt' 'b.txt' | xargs -t -n1 wc

# eq. to
wc a.txt
wc b.txt

# Similarly, `-L 1` means use at most 1 nonblank input line per command line
cat a.txt | xargs -t -L1 wc

# placeholder
cat fruit.txt | xargs -I :FRUIT: echo "buy more: :FRUIT:"

# only delimit input on newline but not space, `-0`
# e.g. there're some directories or files that have spaces in names
ls ~/Library/ | grep 'A.*' | xargs -0 -n1

# combine with find's -print0
find test1/ -type f -print0 | xargs -0 chmod 755

# interactive, `-p`
find . -name "*.backup" -depth 1 -print0 | xargs -p -0 rm

# find total number of lines of all files in current directory
find . -name '*.php' | xargs wc -l

15.25.14. test - Conditionally run 

# -L :: if the file exists and it's a symbolic link
test -L /bin/systemctl || ln -sf /usr/bin/systemctl2 /bin/systemctl

15.25.15. Concatenate String with File

# Append to the beginning
#   - echo returns a newline
echo '<?xml version="1.0"?>' | cat - input.txt > output.txt

# '-' after 'cat' means to take the stdin from last command

# Append to the end
echo 'end line' | cat input.txt - > output.txt

15.25.16. Change Encoding

Change from UCS-2 to UTF-8
iconv -f UCS-2 -t UTF-8 input.xml > output.xml

15.25.17. Comment out every line in a file and save as a new file

Add '# ' in front of every line jail.conf is the source file, and jail.local is the destination file. After it's run, jail.conf is not affected. 

awk '{ printf "# "; print; }' /etc/fail2ban/jail.conf | sudo tee /etc/fail2ban/jail.local

15.25.18. sudo

Install sudo apt-get update && apt-get install -y sudo less Run sudo interactive sudo -i

15.25.19. SHA, salt hashing image:lucee:salt

./luceehashing.sh UIloginPassword salt 

#!/bin/bash

SHA_ALGORITHM=256
SHA_COUNT=5

LUCEE_PASSWORD=${1:-"_"}
LUCEE_SALT=${2:-"_"}

if [ $LUCEE_PASSWORD == "_" ]; then
  LUCEE_PASSWORD=topsecret
fi
if [ $LUCEE_SALT == "_" ]; then
  LUCEE_SALT=$(uuidgen | tr a-z A-Z)
fi

COUNT=1
LUCEE_HASH=$(echo -n "${LUCEE_PASSWORD}:${LUCEE_SALT}" | shasum -a $SHA_ALGORITHM | cut -f1 -d' ')
while [ $COUNT -lt $SHA_COUNT ]; do
  LUCEE_HASH=$(echo -n $LUCEE_HASH | shasum -a $SHA_ALGORITHM | cut -f1 -d' ')
  COUNT=$((COUNT + 1))
done

echo "Lucee Admin Values"
echo "  hspw = $LUCEE_HASH"
echo "  salt = $LUCEE_SALT"

15.25.20. byobu

Create multiple terminals and switch between. apt-get install byobu

15.25.21. Command line web browser w3m

Ubuntu 

apt-get install w3m
# S-Q to quit

15.25.22. set bash:set

# shell
set -ex; command1; command2; ...

# bash script
set -ex
command1
command2
-x
enable a mode of the shell where all executed commands are printed to the terminal
-e
under common situation, exit when there's an error (stop at first error)

15.25.23. Calendar

cal
show current month
cal 12 2020
show December, 2020
cal -y 2020
show all months in 2020
cal -y
current year

15.25.24. Calculator

scale=10
change precision to 10 decimal points

15.25.25. expr 

expr 1+1
# have to separate the values

expr 1 + 1

# have to escape some characters
expre 1122 \* 3344

15.25.26. unit converter

  • interactive
  • units '1 meter' 'seconds'

15.25.27. shutdown

  • /sbin/shutdown
  • power off and restart
  • halt (power off)

15.26. Cheat

sudo apt-get update && sudo apt-get upgrade
sudo apt-get install python-pip
sudo pip install cheat
cheat -v

Modify ~/.bashrc 

# If not running interactively, don't do anything
case $- in
  *i*) ;;
    *) return;;
esac

export EDITOR="/usr/bin/nano"
export CHEATCOLORS=true

# don't put duplicate lines or lines starting with space in the history.
HISTCONTROL=ignoreboth

Add to autocompletion 

cd /etc/bash_completion.d/
# sudo wget https://raw.githubusercontent.com/chrisallenlane/cheat/master/cheat/autocompletion/cheat.bash
# cheat.bash is as follows
function _cheat_autocomplete {
    sheets=$(cheat -l | cut -d' ' -f1)
    COMPREPLY=()
    if [ $COMP_CWORD = 1 ]; then
        COMPREPLY=(`compgen -W "$sheets" -- $2`)
    fi
}

complete -F _cheat_autocomplete cheat

cheat -d lists the directories that cheat sheets should be saved. One is ~/.cheat, the other is system Each cheat sheet is a file, like ~/.cheat/ping 

;; # some comments
ping -c 15 www.example.com

;; # another usage
ping ...

cheat -s packets search all cheat sheets that has `packets` in commands and description. Also return cheat sheet name

cheat ping show the `ping` cheat sheet

15.27. Makefile

Install in Ubuntu
sudo apt-get update && sudo apt-get install make
Docs
https://www.gnu.org/software/make/manual/make.html

Create a Makefile in current folder 

include env_make
# include a file which defines some variables

NAME=myimagename
VERSION=0.1.0
REPO=mydockerreponame
# define some variables

# escape $ in string variable with $$, say password is $abc$ 
password=$$abc$$

# trylogin:
#     sshpass -p '$(password)' ssh -o StrictHostKeyChecking=no root@123.123.123.123

# Define dynamic variable
filename := file_$(shell date +%FT%T%Z).log

.PHONY: default build
# This tells make to not look elsewhere for commands default and build
# Instead, run default and build as they are defined in this Makefile
# Usually include all commands in this Makefile
# And all file or directory names that exist in the directory that Makefile is in

# command in Makefile is called target. Syntax of Makefile is
# target ...: prerequisites ...
#     recipe
#     ...
#     ...

# A prerequisite is a file/command. It means to include a file

# A recipe is an action (command). Every recipe line has to start with a tab character

default:
    @echo "Some description"
# If you run `make` in current folder without specifying which command, the first command will be run

# echo without @ prints
# echo "Some description"
# while @echo prints
# Some description

all: build
# the all command runs the build command

build:
    sudo docker build -t $(NAME):$(VERSION) --rm .

push:
    docker push $(NAME)/$(REPO):$(VERSION)

release: build
    make push -e VERSION=$(VERSION)
# `make release` runs `build` first, then run `push` with a parameter
# can't put push as a prerequisite after `build` because it has to be run using a parameter

testecho:
    @echo $$FOO
# make testecho FOO="a b c"
# actually @echo $(foo) then ~make testecho foo="a b c"~

testsub:
    @echo otheraction $(filter-out $@,$(MAKECMDGOALS))

# make testsub a b c
# $@ is the target name, which is testsub
# $(MAKECMDGOALS) is the list of targets, which is testsub a b c
# This method requires you to put these 2 lines to the end of Makefile (remove leading #'s)
# %:
#    @:
# 
# %: is a target which matches every target
# @: is a recipe in which @ is the same as @echo. : means do nothing

# Escaping $ might be needed for running sub command
start:
    docker rm -f $$(docker ps -a -q)

# - in front of command means ignore the exit status of the command so that a non-zero exit would not stop the recipe
runrecipeinmiddle:
    - echo do something
    - $(MAKE) build

testpwd:
    cd ~/abc; echo `pwd`; # ~/abc
    echo $(PWD); # the path where `make testpwd` is run
Makefile
make your-recipe
Makefile.test
make -f Makefile.test your-recipe
Makefile.live
make -f Makefile.live your-recipe

Be careful to close single quotes in @echo

15.27.1. Characters to escape

https://www.cmcrossroads.com/article/gnu-make-escaping-walk-wild-side 

Escape # as \#
Escape $ as $$

15.27.2. Options

15.27.2.1. -C dir, --directory=dir
phpcs:
    $(MAKE) --directory api phpcs # make -f ./api/Makefile phpcs

15.28. Nano

  • Meta key (M-) from Mac is Escape
  • C-o
  • C-x
  • M-u
  • M-e
  • C-c
    Show line number when open
    nano -c filename
  • C-u
  • C-w C-v or C-_ C-v
  • Select and copy, paste
    • S-M-a or C-6 to set mark, move cursor to selct
    • S-M-6 or M-6 to copy or use C-k to cut text
    • C-u to paste

    • To delete all content in a file

      • Go to the beginning of the file, set mark C-6
      • Go to the end of the file C-w C-v
      or S-M-t
      but Zoom may take over this shortcut and do screenshot instead. Close Zoom to avoid its shortcut
      • Delete Backspace
  • Esc, release, then $ (shift+4) or M-S-4
  • C-S-c have to ensure visual word wrap is on for multiple lines
  • M-S-q
  • https://nano-editor.org/dist/latest/cheatsheet.html
  • MacOS iTerm2
    Enter Copy Mode
    ⇧ ⌘ c
    Toggle selection by character
    Space
    (no term)
    Move cursor to select
    Copy
    y
    Stop selecting
    C-Space
    Paste
    ⌘ v

15.28.1. TS: File is being edited by root

SSH connection failed/interrupted when editing a file, nano saved a snapshot

  • FILE.txt.save or .FILE.txt.swp
  • Delete the snapshot/lock file

15.29. Vim

Interactive tutorial
https://www.openvim.com/

15.29.1. Normal Mode

15.29.1.1. Move Cursor
  • h, j, k, l
  • start of next word
  • end of the word
  • beginning of the word (previous word beginng)
  • {number}{h, j, k, l, w, e, b}
15.29.1.2. Copy, Cut, Paste
dd
delete a line and add it to the default register
p
paste below the current lines
S-p
paste above the current line
ddkP
move up a line
ddp
move down a line
15.29.1.3. Replace
ciw
change inner word. Delete word at cursor and go to Insert Mode
15.29.1.4. :cq :: exit with an error and the file is not saved

15.29.2. Insert Mode

30i- Esc
insert - 30 times and jump back to normal mode

15.30. Ubuntu

15.30.1. Gnome

Logout
gnome-session-quit
Logout gnome session in SSH remote session (haven't used it myself)
env DISPLAY=:0.0 gnome-session-quit --logout --force
Reboot in SSH remot session
env DISPLAY=:0.0 gnome-seesion-quit --reboot

15.30.2. Shortcuts

Most of system shortcuts
Settings > Devices > Keyboard
Search a shortcut containing s
gsettings list-recursively | grep -i \>s\'
(no term)
e.g. it returns this org.gnome.shell.extensions.screenshot-window-sizer cycle-screenshot-sizes ['<Alt><Control>s']
(no term)
sudo apt install dconf-editor && dconf-editor and go to that folder
(no term)
Turn off Use default value and in Custom value change it from ['<Alt><Control>s'] to []
Terminal
C-M-t
Copy from Terminal (GNOME)
C-S-c
Open favorite app 1
Super-1
Show All Applications
Super-a
Show Notification tray
Super-m
Run commands
M-<F2>
(no term)
Switch workspaces C-M-Up or C-M-Down
(no term)
Switch apps (open App Switcher) M-Tab
(no term)
Switch instances/windows for the current app M-`
Quit app in App Switcher
M-q
Close application window
C-q or M-F4
(no term)
Drag and drop with C-S to create symlinks

15.30.3. Setting

https://linuxconfig.org/things-to-do-after-installing-ubuntu-18-04-bionic-beaver-linux

  • Keyboard repeat rate

Settings > Universal Access > Typing > Repeat Keys > On, lower speed faster rate

  • Remote Desktop from Windows RDP

http://c-nergy.be/products.html 

chmod +x  ~/Downloads/Std-Xrdp-Install-0.5.1.sh
./Std-Xrdp-Install-0.5.1.sh -s yes -g yes

sudo reboot

# try remote from Windows
  • Install Pinyin
    • sudo apt install ibus-pinyin
    • Settings > Languages and Region > Manage installed Languages, reboot or select IBus as Keyboard input method system, add Chinese Simplified then reboot
    • Settings > Languages and Region > add Chinese (Intelligent Pinyin) as a new Input Sources
  • Software & Updates > Ubuntu Software > Download from > Other > Select Best Server
  • gnome-tweak-tool
    • sudo add-apt-repository universe sudo apt install gnome-tweak-tool
    • Open Tweaks app
  • prepare to install extensions that come with the package repo
    • Log back in and go to Tweak Tool > Extensions
  • GDebi
    • By default, Ubuntu Software Center does not install dependencies when app .deb is installed
    • sudo apt install gdebi
    • e.g. download deb from Chrome and in Files app, right click > Properties > Open with GDebi Package Installer, close
    • sudo gdebi google-chrome-stable_current_amd64.deb
    • To remove a package installed from gdebi, apt and dpkg commands using purge option as shown: sudo apt purge google-chrome-stable sudo apt-get purge teamviewer sudo dpkg -purge teamviewer
  • Snap
    • Snap is built-in in Ubuntu 18.04
    • Apps installed by Snap are almost independent. Different versions of the same app can co-exist
    • Before developers have to come up packages for each distro and each version of distro. Now, one snap one distro.
    • Snap Daemon can be used in distros other than Ubuntu
    • Snaps are automatically updated (4 times a day)
    • Snap is installed globally for all users once and each user stores separate userdata
    • https://snapcraft.io/store
    • sudo apt install snapd
    • Find Snap package snap find vlc
    • sudo snap install vlc
    • snap list
    • snap changes
    • Upgrade sudo snap refresh vlc
    • sudo snap refresh --list
    • Rever a recent update sudo snap revert vlc
    • sudo snap remove vlc
    • stable, candidate, beta, edge
      • sudo snap refresh vlc --channel=edge
    • Download and install it offline later
      snap download vlc
      download .assert and .snap file
      (no term)
      snap ack vlc.assert
      (no term)
      snap install vlc.snap
  • sudo apt install ubuntu-restricted-extras
  • Intel graphic card screen tearing under X11

Reference 

sudo mkdir -p /etc/X11/xorg.conf.d/
sudo nano /etc/X11/xorg.conf.d/20-intel.conf

Section "Device"
   Identifier  "Intel Graphics"
   Driver      "intel"
   Option      "TearFree"    "true"
EndSection

Reboot

15.30.4. Dual Boot with Win 10

Turn off fast start up
Control Panel > Hardware and Sound > Power Options > Choose what the power buttons do > uncheck Turn on fast startup
May need to disable Secure Boot
enable means BIOS will prevent un-authorized OS be loaded.
  • Win + S to search advanced startup options and click Restart Now
  • select TroubleShoot > Advanced Options > UEFI Firmware Option Settings > Restart
  • BIOS settings will open, disable Secure Boot. You may need to Turn Legacy Support On/Off
(no term)
Determine if it's BIOS or EFI mode
  • Run msinfo32 and see if BIOS Mode is UEFI
    BIOS mode
    boosts by reading the first sector on hard disk and executing it; this boot sector in turn locates and runs additioinal code. It uses the Master Boot Record (MBR) partition table which is very limiting because of space (no more than 2TB in size per partition) and partitions(more than 4 primary partitions) constraints.
    EFI mode
    boots by loading EFI program files (with .efi extension) from a partition on the hard disk uses the GUID partition table (GPT) offering 64-bit entries in its table which dramatically extends the support for size possibilites.
  • Ubuntu has to be installed using the same mode. If you see black screen with options when you boot using Ubuntu's bootable USB, then it shows Ubuntu is booted from EFI.
Create free space on Windows for Ubuntu
Win + S to search Create and Format Hard Disk Partitions > Shrink Partition, so that you will have Unallocated space in partitions. You will need 2 unallocated spaces. One for root and one for /home in Ubuntu
(no term)
Create a bootable USB stick on Windows
Boot from USB
Win + S to search Advanced Startup Options > Restart Now > Boot from USB
(no term)
Try Ubuntu without installing > open gparted, right click on unallocated to create partition
  • Create as: Primary Partition, File system: ext4, Label: ROOT and HOME, partition name: root and home,
(no term)
Open Install on the desktop
(no term)
Normal installation and Install third-party software for graphics and Wi-Fi hardware, MP3 and other media
Installation type
Something else
change on a device name
Use as: Ext4 journaling file system, Mount point: / and /home, Format the partition.
Device for boot loader installation
If boot mode is BIOS, select the whole device name e.g. /dev/sda. If it's UEFI mode, choose the partition. Go to gparted and see which partition has value boot in column 'Flags'. e.g. /dev/sdap1

15.30.5. Remmina to Win 10

  • Enable RDP on Win 10
  • Remina change connection Color depth to GFX RFX or High color (16 bpp)
  • By default, use Right Ctrl to toggle Grab all keyboard and then Win key can be passed to Win 10
    • Remina > Preferences > Keyboard > Grab keyboard

15.30.6. Citrix

  • Better to install Workspace app for Linux (Full Package 64bit) instead of Citrix Receiver for Linux

TS: Cannot connect to "Your Connection Name" No such file or directory. Verify your connection settings and try again. 

cd /opt/Citrix/ICAClient/keystore/
sudo cp -ar cacerts cacerts-bk
sudo rm -rf cacerts
sudo ln -s /etc/ssl/certs cacerts

15.30.7. Keychain

  • Search password to get Passwords and Keys
  • App usually use Passwords > Login
  • phpStorm uses IntelliJ Platform DB — hash-id-get-it-from-proj-folder/idea/dataSources.xml

15.30.8. Sync date time

# 18.04
# see if System clock synchronized is yes
timedatectl status

# check time difference between the local system and the internet time server, and sync if necessary
sudo chronyd -q

# to install
sudo apt install chrony

15.30.9. Maintenance

sudo apt autoremove --dry-run
delete any unused dependencies
sudo du -sh /var/cache/apt
APT keeps previously downloaded and installed packages even after they've been uninstalled
  • Remove only the outdated packages sudo apt autoclean --dry-run
  • Clean it entirely sudo apt clean --dry-run

15.31. Ubuntu Personal Package Archive - PPA

Add a LibreOffice package from PPA to Software Sources
sudo add-apt-repository ppa:libreoffice/ppa, then sudo apt-get update
Remove the package source
sudo add-apt-repository --remove ppa:libreoffice/ppa, apt-get update
Remove it and downgrade back to previous version before using PPA
sudo apt-get install ppa-purge, sudo ppa-purge ppa:libreoffice/ppa

15.32. Production Server Setup

linux:user
Add a normal user, add the user to sudo group
linux:ssh
Add ssh public key to the new user's .ssh folder
linux:ssh:disallow_root
Disallow SSH in as root
linux:sudo:nopassword
require no password for non-root users with sudo privilige/group
linux:firewall
configure firewall, open ports
linux:timezone
setup timezone and sync time

16. Domain Server, DNS Server, SSL/https

See linux:dns

16.1. DNS records: A AAAA ANAME CNAME TXT ALIAS URL

Hostname
a sub domain. e.g. www @
A record
redirect hostname.domainname.com to an IP address. For the barebone domain name, the hostname is @
wildcard
hostname * or *.asubdomain
AAAA records
same as A records but they are IPv6. Nowadays, both AAAA and A are needed
ANAME
It's like CNAME. Name (from source domain) could be @. The value has to be another (target) apex domain. DNS resolves the target fully qualified domain name (FQDN) to an IP or multiple and also synthesize A records of the domain that points to the IP(s) of the FQDN. Since the target FQDN's IP is returned on the first lookup, it's faster than ALIAS record.
CNAME records
point another subdomain (hostname) to an A record (e.g. hostname.domainname.com). It should only be used when there are no other records on that name. Don't use the bare domain in CNAME on the left hand side.
ALIAS records
same as CNAME but it can coexist with other records on that name. It resolves the target domain to one or more A records. ALIAS's name can be (from source domain) anything, the value can be (target) any other external domain.
URL records
redirect the name to the target name using HTTP 301 status code
  • It redirects the name to a destination. The URL record is simple and effective way to apply a redirect for a name to another name, e.g. to redirect www.example.com to example.com
TXT record
Google Search Console may add TXT @ google-site-verification=[code] TTL 1 Hour to verify site ownership.
A, CNAME, ALIAS records
A name must resolve to an IP, the CNAME and ALIAS record must point to a name
(no term)
General rule
  • use an A record if you manage what IP addresses are assigned to a particular machine or if the IP are fixed (this is the most common case)
  • use a CNAME record if you want to alias a name to another name, and you don't need other records (such as MX records for emails) for the same name
  • use an ALIAS record if you are trying to alias the root domain (apex zone) or if you need other records for the same name
  • use the URL record if you want the name to redirect (change address) instead of resolving to a destination.
(no term)
On bare/apex domain, usually A, ALIAS or ANAME record can be used. Only DNS Made Easy supports ANAME and DNSimple and others support ALIAS. ANAME and ALIAS are not industry standards

16.2. MX record dns:mx

HOSTNAME
e.g. use @ for barebone
Mail Providers Mail Server (or Value)
always end with "." e.g. alt4.aspmx.l.google.com.
Priority
the smaller the more important

Google G Suite requires TTL to be 1 hour (3600). Priority might be different but aspmx.l.google.com. must be the highest Priority Mail Server 1 ASPMX.L.GOOGLE.COM. 5 ALT1.ASPMX.L.GOOGLE.COM. 5 ALT2.ASPMX.L.GOOGLE.COM. 10 ALT3.ASPMX.L.GOOGLE.COM. 10 ALT4.ASPMX.L.GOOGLE.COM.

Google G Suite Toolbox Check MX

16.3. SPF record dns:spf

SPF record is a TXT record which has 0 or more ordered mechanisms all | ip4 | ip6 | a | mx | ptr | exists | include

Mechanism can be prefixed with 1 of 4 qualifiers. A qualifier means if the machanism matches the sender's IP, then return a result. If a mechanism matches the sender but no qualifier, then + is used. If no mechanism or modifier matches, then default result is Neutral. 

+ Pass
- Fail
~ SoftFail
? Neutral

# Google G Suite SPF record
v=spf1 include:_spf.google.com ~all

# Add domain's A, MX records to the match list and always allow.
# Include a domain to the match list.
# Soft fail for all other senders IP or domains.
v=spf1 +a +mx include:_spf.google.com ip4:1.2.3.4 ip4:1.2.3.5 ip4:1.2.3.6 ~all
  • Mechanism all matches all senders' IP's, and it's usually at the end:
    • -all fail for all
    • ~all soft fail for all
    • +all allows all senders
  • Mechanism mx adds domain's MX details to the match list.
  • Mechanism a adds domain's A records to the match list

16.4. DKIM record dns:dkim

  • One domain can have multiple DKIM records
  • Mail server signs the email with the private key and the receiving mail server uses the public key in the domain's DNS info to verify the signature
  • Name: s1._domainkey Value: s1.domainkey.xxx.xx.sendgrid.net
    On the mail server e.g. SendGrid
    TXT s1._domainkey the-domain-key-value see below
  • TXT customer._domainkey k=rsa; p=xxxx for a root domain, for subdomain TXT customer._domainkey.e k=rsa; p=xxxx
  • When the email is sent from an SMTP mail server e.g. SendGrid, the receiver will have Sender Domain (the From: email address), Envelope Domain (e.g. Zoho Campaign mail server is zcsend.net) and DKIM Domain
  • Envelope Domain is the domain used to receive bounce error messages (e.g. bounce rate) from recipient systems and I think it's specified in the sent email as a header Return-Path
  • DKIM Domain is specified as a header in the sent email using DKIM-Signature
    • v=1; a=rsa-sha256;d=abc.com;s=customer;c=relaxed/relaxed;q=dns/txt;t=1554220118;h=x-fbl:mime-version:date:message-id:content-type:list-unsubscribe:from:to:subject; bh=XXXHAHSED; b=XXXHAHSED
    • the sign algorithm
    • the signing domain which is also the DKIM Domain
    • the selector used to find DKIM public key info. To match the domain's DKIM record e.g. as in customer._domainkey
    • the canonical algorithm(s) for header and body
    • the default query method
    • signature timestamp
    • the list of signed header fields, repeated for fields that occur multiple times. Basically saying this signature verifies these fields are true
    • body hash
    • the actual digital signature of the contents (headers and body) of the mail message
  • If Sender Domain and Envelope Domain do not match, then SPF is not aligned. If Envelope Domain is a subdomain of Sender Domain, they are aligned
  • If dns:dmarc has aspf=r will reject if SPF is not aligned. But it's common to have SPF not aligned
  • Next is to check if DKIM Domain is aligned with Sender Domain. If they don't match, DMARC will definitely fail. So you will have to have a different DKIM record setup for each Sender Domain

16.5. SOA record

The SOA record stores information about:

  • the name of the server that supplied the data for the zone;
  • the administrator of the zone; e.g. b.c.com is actually the email address b@c.com
  • the current version of the data file;
  • the number of seconds a secondary name server should wait before checking for updates;
  • the number of seconds a secondary name server should wait before retrying a failed zone transfer;
  • the maximum number of seconds that a secondary name server can use data before it must either be refreshed or expire;
  • and a default number of seconds for the time-to-live file on resource records.

A DNS zone is the part of a domain for which an individual DNS server is responsible. Each zone contains a single SOA record.

e.g. ns2.a.com. b.c.com. 2018012804 10800 3600 604800 10800

16.6. redirect.center, redirect.name

If URL record is not available, use CNAME and redirect.center and redirect.name Instructions: https://milanaryal.com/domain-redirecting-and-url-forwarding-with-a-simple-dns-record/

redirect.center is especially good for redirect subdomain www.abc.com to google.com/path if the domain host doesn't support Domain Forwarding with path.

To redirect naked domain, use Forward Domain. 

;; redirect google.example.com to google.com
google.example.com IN CNAME google.com.redirect.center

;; redirect google.example.com to google.com with a 302 status code
google.example.com IN CNAME google.com.opts-statuscode-302.redirect.center

;; If a user try to visit www.oldwebsite.com/about/ will redirect to path www.newwebsite.com/about/:
www.oldwebsite.com  IN  CNAME  www.newwebsite.com.opts-uri.redirect.center

;; opts-statuscode-{code} :: HTTP Status Code to be used in the redirect. 302, HTTP Status Code
;; opts-uri :: Append URI (if any) to the target URL
;; opts-slash
;; jobs.my-domain.com to http://www.my-domain.com/jobs
;; jobs IN CNAME to www.my-domain.com.opts-slash.jobs.redirect.center
;; multiple path components, jobs.my-domain.com to www.my-domain.com/jobs/xyz
;; jobs IN CNAME to www.my-domain.com.opts-slash.jobs.opts-slash.xyz.redirect.center

;; Use redirect.name instead of readirect.center
;; redirect google.example.com to google.com
google.example.com            IN  CNAME  alias.redirect.name
_redirect.google.example.com  IN  TXT    Redirects to https://www.google.com

;; redirect google.example.com to google.com with options
google.example.com            IN  CNAME  alias.redirect.name
_redirect.google.example.com  IN  TXT    Redirects from /* to https://www.google.com/*

;; Redirects to [target], where target is the target URL
;; Redirects from [path] to [target], where path is a path to match on the hostname
;; Redirects permanently to [target], where permanently redirects with a 301 status code (defaults to 302 otherwise)

;; redirect www.my-domain.com to https://my-domain.com
www.my-domain.com IN CNAME my-domain.com.opts-https.redirect.center

16.7. Domain Forwarding

Naked domain forwarding Forward abc.com to http://google.com abc.com/xyz will become google.com/xyz in the address bar

Forward abc.com to http://google.com/xyz Some domain host and registration service might support that. abc.com/ijk will become google.com/xyz/ijk in the address bar

Subdomain forwarding can be done as well.

Domain registrar might require to purchase SSL with domain in order to forward https://olddomain.com to https://newdomain.com

Mask is http://oldomain.com continues to appear as oldomain.com in address bar but it actually points to newdomain.com

16.8. Domain Transfer

  • Make sure domain Admin contact email is valid. Unlock domain and remove privacy. Ask for auth code
  • Go to new registrar, in order to maintain the old nameservers, GoDaddy requires to setup DNS for the domain before request for domain transfer
  • Purchase domain transfer and enter auth code. .CA domains can be transferred immediately
  • On new registrar, create a DNS template and input old DNS records
  • At the end, change the nameservers to the new registrar and apply the DNS template with old records

16.9. Addon domain

It's an addtional domain that the system stores as a subdomain abc.com is registered in GoDaddy and will be added as an addon domain on abc.xyz.com. Change nameservers to the domain xyz.com host say ns.googlehosting.com ns2.googlehosting.com Then in googlehosting, set it up as an addon domain :: abc.com > subdomain abc.xyz.com, then specify the document root

abc.com will appear as parked on subdomain abc.xyz.com on the googlehosting

An addon domain has to be mapped to a main domain's subdomain. In this case, the subdomain is abc.xyz.com

Source domain is abc.com registered in GoDaddy and nameservers are later changed to destination domain's Google nameservers. Destination domain is abc.xyz.com and the root domain xyz.com is registered and DNS managed in Google.

You don't need to first create a subdomain and then create Addon domain. It will ask you to create a subdomain in the destination domain and point to a directory when you create an addon domain.

Source domain can be a subdomain e.g. s1.abc.com, change A record for s1.abc.com at GoDaddy to point to the destination's server IP.

16.10. Park domain

  • To park a domain means to make the domain as idle. Nameservers will be set to Default e.g. in GoDaddy
  • To park multiple domains abc.net, abc.info to the main domain abc.com

16.11. Domain Registrars and Tools

  • GoDaddy

    • Delegate Access godaddy:delegate access

      • Source GoDaddy account s@a.ca gives limited access to another GoDaddy account d@a.ca
      • Source account > Account Settings > Delegate Access > choose to send to b@a.ca
      • The person clicks an invite in mailbox b@a.ca, then logs on to d@a.ca. Then GoDaddy account d@a.ca can access GoDaddy account s@a.ca
  • Hover.com
  • search for the right domain using your words

16.12. DNS History, IP location, Website Hosting Finder

DNS history
https://securitytrails.com/dns-trails
See all DNS records
https://www.ultratools.com/tools/dnsLookup
Lookup domain registrar and nameservers use whois
https://mxtoolbox.com/Whois.aspx
Find IP location
https://www.iplocation.net/
DNS propagation
https://www.whatsmydns.net/
(no term)
Find hosting of a website

16.13. AXFR and Find subdomains

Without DNS Zone file export, it's almost impossible to find all subdomains and their DNS records. The following can be used:

  • Brute force dig a.x.com dig b.x.com
  • Pentest-tools.com - Find Subdomains">not guaranteed to show all subdomains
  • If AXFR is enabled on DNS server and the IP from which this command is run is added as a secondary IP address to the target domain, the run host -l x.com or dig @ns1.DomainsSOA.com x.com axfr
  • Use Google search results to filter out known subdomains site:x.com -"www.x.com" -"a.x.com" -"b.x.com"

DNS servers host are known as zones. DNS zone transfer is to replicate db changes across several DNS servers. AXFR is referred as DNS zone transfer because it's the protocol used during a DNS zone transfer.

Initiate an AXFR zone-transfer request is simple and can be used by hackers. The below replicate x.com DNS records to the first nameserver. dig @ns1.DomainSOA.com x.com axfr

16.14. AutoSSL in WHM/cPanel autossl:whm

  • WHM and cPanel provides free SSL certificates for VPS and Dedicated server users
  • It uses SHA256 TLS v1 and expires every 90 days. TLS v1 must be disabled for eCommerce site by June 30, 2018. For non-eCommerce site, it's ok
  • AutoSSL requires Domain Control Validation (DCV) - the domain has to have NS or A records point to the server that AutoSSL is generated. Otherwise, it will not do anything, not even creating .well-known folder
  • First is to enable AutoSSL for WHM users. AutoSSL will check all root domains under the user account
  • If you enable a user and the user has 1 root domain and several addon domains and subdomains, all will have SSL
  • AutoSSL will include certificates for www. domains. e.g. www.yourrootdomain.com, www.yoursub.yourrootdomain.com
  • yoursub.yourrootdomain.com was set up by you and you didn't setup www.yoursub.yourrootdomain.com
  • Use hosting nameservers in domain registrar. Log in WHM as root. Go to Manage AutoSSL.
    Providers
    cPanel (powered by Comodo)
    Options
    Check all options.
    "Allow AutoSSL to replace invalid or expiring non-AutoSSL certificates"
    enable this to allow AutoSSL to replace certs that the AutoSSL system did not issue e.g. you import purchased cert. Enable this will overwrite the imported cert with AutoSSL certificates.
    Manage Users
    Enable AutoSSL for relevant users
    (no term)
    Click on Run AutoSSL for All Users
    (no term)
    Check Logs
  • SSL certificates should be now installed
  • http://www.inmotionhosting.com/support/website/cpanel/add-delete-autossl-cpanel
  • A directory .well-known will be created in sub website directory if the parent website directory has AutoSSL

16.15. Install SSL

  • See SSL Test Tools
  • certificate starts with --BEGIN CERTIFICATE--, file extension can be .pem, .crt
    • Fingerprint of a certificate
      SHA-256
      openssl x509 -noout -fingerprint -sha256 -inform pem -in your.pem
      SHA-1
      openssl x509 -noout -fingerprint -sha1 -inform pem -in your.pem
      MD5
      openssl x509 -noout -fingerprint -md5 -inform pem -in your.pem
  • private key starts with --BEGIN RSA PRIVATE KEY--
  • Certificate Chain or Chain of trust: end-user, intermediate and root certificate
    • Certificate Authorities issue SSL/digital certificate
      • certify the ownership of a public key (CSR) which is generated locally
    • Install SSL certificate issued from CA1, this is called the end-user certificate
      • this certificate basically vouches for the CSR which is a URL and its registration info
    • CA1 utilizes a certificate issued by CA2 and CA2 utilizes a certificate issued by CA3
    • CA2's certificate is an intermediate CA
    • CA3 is a root CA and its certificate is directly embeded in the web browser and is called root certificate since its CA is trusted by the browser

  • Generate a private key myserver.key with 2048 bits, then generate a CSR (Certificate Signing Request) file myserver.csr using that private key and answering some registration questions. The CSR contains the answers linux:ssl:server.csr.cnf and a public key 

    openssl req –new –newkey rsa:2048 –nodes –keyout myserver.key –out myserver.csr

# Prompt for Common Name
# example.com for a single domain
# *.example.com for wildcard certificate
  • Submit CSR to SSL authority (CA) to get .crt and use myserver.key generated above and make them root user only
  • Generate self-signed certificate

    • docker-compose.yml 

      ports:
  - "32012:80"
  - "443:443"
    • Create a Root SSL Certificate for domain localhost
      • openssl req -newkey rsa:4096 -nodes -keyout rootCA.key -x509 -days 5000 -out rootCA.pem
    • create a new private key and also certificate request (CR)
      -x509
      create a self signed certificate rather than a cerficiate request (CR)
      -nodes
      create private key without providing passphrase which is at least 4 letters long
      (no term)
      Enter random values for the prompt
      • Optionaly, you can create a certificate with an existing key
      (no term)
      openssl req -new -nodes -key rootCA.key -x509 -days 1024 -out rootCA.pem
      (no term)
      -new with -key creates a CR with an existing key
      • Combine private key and certificate into a .p12 (pkcs12, pfx format) file
      (no term)
      openssl pkcs12 -inkey rootCA.key -in rootCA.pem -export -out rootCA.p12
      (no term)
      Validate openssl pkcs12 -in rootCA.p12 -noout -info
      • Add rootCA.p12 if Chrome supports it. Windows can import .p12 file
      (no term)
      On Chrome, Settings > Manage certificates > Import under Trusted Root Certification Authorities
      (no term)
      After that, select the certificate with name Internet Widigits Pty Ltd. click Advanced and select all Certificate purposes
      (no term)
      In Ubuntu, Chrome only supports pkcs7 (.p7b file). So just add rootCA.pem to Chrome as Authorities
      (no term)
      Without adding to Chrome, https://localhost only works in InCognito mode.

    • Now create domain certificate using the root SSL certificate 

      [req]
default_bits = 4096
prompt = no
default_md = sha256
distinguished_name = dn

[dn]
C=US
ST=RandomState
L=RandomCity
O=RandomOrganization
OU=RandomOrganizationUnit
emailAddress=hello@example.com
CN = localhost

      CN is common name which is a FQDN

      • Create v3.ext for creating X509 v3 certificate
      authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = localhost
      • Now create a certificate key server.key for localhost and also create a certificate signing request (CSR) server.csr using server.csr.cnf

      openssl req -new -nodes -out server.csr -newkey rsa:4096 -keyout server.key -config <( cat server.csr.cnf )

      • Genereate the domain certificate file server.crt using v3.ext that is created earlier

      openssl x509 -req -in server.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out server.crt -days 5000 -extfile v3.ext

      • Some read and verify commands
      # Read and verify a certificate
openssl x509 -text -noout -in server.crt

# Read and verify a key
openssl rsa -check -in server.key

# Read and verify a CSR
openssl req -text -noout -verify -in server.scr

# Read and verify a PKCS#12 file .pfx or .p12
openssl pkcs12 -info -in rootCA.p12 -noout
  • Combine certificates and install

16.16. SSL Testing Tools

17. Docker

17.1. Version, Info, Installation

Brief
docker --version
Server and Client
sudo docker version
Full
sudo docker info
(no term)
"Docker for Windows" needs Windows Hyper-V enabled. Command prompt is admin
(no term)
After installation, make sure linux:firewall DEFAULT_FORWARD_POLICY="ACCEPT"
(no term)
Use systemctl for managing Docker service

17.1.1. Without sudo

Users in user group docker can run docker commands without sudo. 

# replace your username with ${USER}
sudo usermod -aG docker ${USER}

# to apply the new group membership, you can log out and back in, or run this
su - ${USER}

# print name (-n, --name) instead of a number and show group IDs (-G)
id -nG

17.1.2. Install on Digital Ocean

https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-16-04 

sudo apt update

# Add GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# Add Docker repo to APT sources so that Docker is the most up to date
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update

# query the APT cache, it will show docker-ce is from the docker repo
apt-cache policy docker-ce
sudo apt-get install -y docker-ce

# check if Docker is running
sudo systemctl status docker

https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04 

sudo apt update

# install prerequisite packages which let apt use packages over HTTPS
sudo apt install apt-transport-https ca-certificates curl software-properties-common

# add GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# add Docker repository to APT sources:
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"

# make sure to install Docker from the Docker repo instead of the default Ubuntu repo:
apt-cache policy docker-ce

sudo apt install docker-ce

# check status
sudo systemctl status docker

# add your username to docker group to avoid typing sudo
sudo usermod -aG docker ${USER}

# logout or type the following to apply the new group membership
su - ${USER}

# confirm your username is now added to the docker group
id -nG

Install docker:docker compose:ubuntu

17.2. Command Help docker docker-subcommand --help

17.3. Images

List all images
docker images
List all dangling images which can be removed
docker images -f "dangling=true"
Remove all dangling images (-q is to return IDs only)
docker rmi $(docker images -f "dangling=true" -q --no-trunc)
Remove all none images
docker rmi $(docker images | grep "none" | awk '/ / { print $3 }')
Download an image
docker pull phusion/baseimage or with a tag docker pull phusion/baseimage:0.9.18
  • docker pull --platform=linux/amd64 mysql:5.7.40
Remove an image
docker rmi imageName
Save an image to tar file
docker save -o /path/docker-image-file.tar docker-image-name
Import a tar to image
docker load -i /path/docker-image-file.tar
Search image in Docker repository
docker search ubuntu
Setting for an image
docker image inspect php:7.0-fpm

Extract Dockerfile from an image. Use one or more of below 

#!/bin/bash
docker history --no-trunc "$1" | \
sed -n -e 's,.*/bin/sh -c #(nop) \(MAINTAINER .*[^ ]\) *0 B,\1,p' | \
head -1
docker inspect --format='{{range $e := .Config.Env}}
ENV {{$e}}
{{end}}{{range $e,$v := .Config.ExposedPorts}}
EXPOSE {{$e}}
{{end}}{{range $e,$v := .Config.Volumes}}
VOLUME {{$e}}
{{end}}{{with .Config.User}}USER {{.}}{{end}}
{{with .Config.WorkingDir}}WORKDIR {{.}}{{end}}
{{with .Config.Entrypoint}}ENTRYPOINT {{json .}}{{end}}
{{with .Config.Cmd}}CMD {{json .}}{{end}}
{{with .Config      .OnBuild}}ONBUILD {{json .}}{{end}}' "$1"

17.3.1. docker build, docker tag

With one Dockerfile in a folder (resource). Run to build
docker build -t your-image-name .
(no term)

Rename image: Change myimage(:latest) to myimage:1.0 

docker tag myimage myimage:1.0

# rename from myname1 to myname2
docker image tag myname1 myname2

# docker image tag eq. docker tag
docker rmi myname1
  • docker tag myimage myimage:1.0
Build an image and tag it with latest and 2.0
docker build -t myimage:latest -t myimage:2.0 .
Multiple Dockerfiles (Dockerfile1, Dockerfile2) in the current folder
docker build -t your-image-name -f Dockerfile2 .
Dockerfile and resource are in different folders (e.g. COPY a file from parent folder in Dockerfile)
docker build -t your-image-name -f /path/Dockerfile.yourname /path/to/docker/
(no term)

Build a lot of times while testing a Dockerfile 

docker build --rm --force-rm --no-cache -t my-image-name .

17.4. run

17.4.1. Interactive mode with tty -it

If there's an error, cannot enable tty mode on non tty input, try -i only

17.4.2. Run an image in a container and keep running in background

docker run --name wordpressdb -v=/vagrant/mydbdump:/tmp/mydbdump \
-v=/mydbdata:/var/lib/mysql \
-e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=wordpress \
-d mysql:5.7.9

Container name: wordpressdb Image name: mysql:5.7.9 Detached mode (in background): -d Insert environment variables: -e

17.4.3. Run an image in a temp container only once, then remove the container

A MySQL container, wordpressdb, created by mysql:5.7.9 image, is already running.

Start up another container using the same image In this temp container, wordpressdb container can be discovered as wpdb (alias). Then import a .sql file to wpdb The temp container is then removed.

Refer to docker:link 

docker run -it --rm \
-v=/vagrant/mydbdump:/tmp/mydbdump \
--link wordpressdb:wpdb mysql:5.7.9 \
sh -c \
'exec mysql -h"$WPDB_PORT_3306_TCP_ADDR" \
-P"$WPDB_PORT_3306_TCP_PORT" \
-uroot -p"$WPDB_ENV_MYSQL_ROOT_PASSWORD" \
wordpress < /tmp/mydbdump/pantheon_db_sql'

17.4.4. Link docker:link

Deprecated and it relies on Docker in default bridge network mode.

17.4.5. –name and –hostname

Specify the container name using –name Inside the container, the container is called –hostname=value docker run --hostname=value OR docker run -h value Combine –name and –hostname as much as you can

17.4.6. –privileged docker:run:privileged

e.g. allow docker container to run docker daemon, allow docker container to access all devices such as file systems.

17.4.7. Port -p 1234:3000

17.4.8. --platform linux/x86_64

  • See dockerfile:from:platform
  • Try to use --platform linux/amd64 instead of linux/x86_64
  • When an image is run, may encounter this warning. Having the platform option changes the host platform arch
    • WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

17.4.9. -v, --volume vs --mount volume

  • https://docs.docker.com/storage/volumes/
  • Bind mount vs named volume
    Bind mount
    no volume is created, direct map host path to container path
    Named volume or data volume
    fully controlled by Docker (Linux VM)
  • See 17.8
  • See 17.11.3.3
  • Dockerfile and docker-compose can use relative path in source. However, docker run cannot. Use linux:makefile

  • For named volumes, both -v and --mount cover all options 

    # -v volumeName:/target/in/container:comma-separated,list,of,options,eg,ro
docker run -d --name devtest -v myvol2:/app nginx:latest

# --mount
# type=bind, volume or tmpfs
# source= or src=
# destination= or dst= or target=
# 
docker run -d --name devtest --mount source=myvol2,target=/app nginx:latest

17.5. Containers cp logs stats start stop restart rm inspect

List containers
docker ps -a
Stop and Start
docker stop containerName docker start containerName
Restart
docker restart containerName
Stop and remove
docker rm -f containerName
Stop and remove all containers
docker rm -f $(docker ps -a -q)
Stop and remove all exited containers
docker rm $(docker ps -qa --no-trunc --filter "status=exited")
Container log
docker logs -f containerName -f to follow log output

17.5.1. docker inspect

  • docker inspect containerName
  • docker:inspect:ip">NetworkSetting > IPAddress or
docker inspect containername | grep -i "IPAddress"
Volume
Config > Volumes and Mounts
Return true or false for container running state
docker inspect -f {{.State.Running}} $CONTAINER_ID

17.5.2. docker events - Show docker system log

17.5.3. docker cp

Copy files from container ghost to docker host
docker cp -L ghost:/usr/src/ghost/config.js ./config.js
Copy files from docker host to container
docker cp -L ./config.js ghost:/usr/src/ghost/config.js
Copy a folder, result /var/www/transfer/web
docker cp web/. ghost:/var/www/transfer/web
Copy a folder, result /var/www/transfer/web
docker cp web/ ghost:/var/www/transfer
-L
copy symbol link
-a
copy file permissions as well as userid:groupid

SRC_PATH specifies a file

  • DEST_PATH does not exist the file is saved to a file created at DEST_PATH
  • DEST_PATH does not exist and ends with / Error condition: the destination directory must exist.
  • DEST_PATH exists and is a file the destination is overwritten with the source file’s contents
  • DEST_PATH exists and is a directory the file is copied into this directory using the basename from SRC_PATH

SRC_PATH specifies a directory

  • DEST_PATH does not exist DEST_PATH is created as a directory and the contents of the source directory are copied into this directory
  • DEST_PATH exists and is a file Error condition: cannot copy a directory to a file
  • DEST_PATH exists and is a directory
    • SRC_PATH does not end with /. (that is: slash followed by dot) the source directory is copied into this directory
    • SRC_PATH does end with /. (that is: slash followed by dot) the content of the source directory is copied into this directory

17.5.4. docker stats

  • CPU and Memory usage for all containers docker stats -a default just shows running containers

Show container names 

docker stats --format "table {{.Name}}\t{{.Container}}\t{{.CPUPerc}}\t{{.MemPerc}}\t{{.MemUsage}}\t{{.PIDs}}"

docker stats --no-stream --format "{{.Name}}: {{.CPUPerc}}"

# other column values
# \t{{.NetIO}}\t{{.BlockIO}}

17.5.5. Save a running container to an image

First exit the container first docker commit -m "added node.js" -a "Author Name" 123456 my/ubuntu-nodejs

Container id :: 123456. get this from docker ps -a New image name :: my/ubuntu-nodejs

17.5.6. Execute commands inside a running container

Login to container with bash open
docker exec -it containerName bash -I
(no term)
Long command (pure-pw only exists in container)

docker exec -it containerName pure-pw useradd bob2 -f longpathto/passwd -m -u ftpuser -d /home/ftpusers/bob2

  • Run multiple commands

docker exec -it containerName bash -c 'cd /var/log; tar -cv ./file.log'

17.6. Network

;; # List networks
docker network ls

;; # Info about a network
docker network inspect the_network_id

;; # Remove networks
;; # returns error when last bridge is removed. It's normal
docker network rm $(docker network ls | grep "bridge" | awk '/ / { print $1 }')

network has active end points. Find the Containers that has Name (mycontainer_name) and EndpointID 

docker network inspect mynetwork_name

;; # you may need to stop and remove the container
docker rm -f mycontainer_name

;; # disconnect the container from a network
docker network disconnect -f mynetwork_name mycontainer_name

;; # run docker-compose down again
docker-compose down -v

17.7. My Network

Refer to CIDR Netmask network:port Personal dev projects use 172.20.x.0/24 Work dev projects use 172.19.x.0/24

Personal dev project port range up to 49151

mysql
31000 to 31999
php
32000 to 32999

Personal dev project

172.20.0.0/24
wordpress, port mysql:31000 php:32000
172.20.1.0/24
drupal 7, port mysql:31001 php:32001
networks:
  app_netsandbox:
    driver: bridge
    ipam:
      driver: default
      config:
      - subnet: 172.20.0.0/24
        gateway: 172.20.0.1
services:
  dbtest:
    networks:
      app_netsandbox:
        ipv4_address: 172.20.0.2

17.8. Volume

17.8.1. Basics

# List all volumes
docker volume ls

# Info about a volume
docker volume inspect the_vol_id

# a volume can be created
# - when using `docker run -v some_volume:/path/to/container`
# - when using docker-compose
# - when using docker volume
docker volume create --name dbdata

# Remove volumes dangling
docker volume rm $(docker volume ls -qf dangling=true)

# Remove a volume
docker volume rm dbdatavolume

17.8.2. Access the named volume by starting another container

On Windows, you may be forced to have named volume because file permissions cannot be set. In this case, you can start a new container to access the volume

Container yourContainerName has named volume dbdata pointed to /var/www/html 

# check what files are in the volume
docker run -ti --rm --volumes-from yourContainerName ubuntu:latest
ll /var/www/html

# backup volume `dbdata` to host backup/backup.bar
docker run --rm --volumes-from yourContainerName -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /var/www/html

# In the backup.tar, the file structure is var/www/html/*

# restore volume for a container
# create a container `bkcontainer` with  `dbdata` and bash in
docker run -v /dbdata --name bkcontainer ubuntu /bin/bash

docker run --rm --volumes-from bkcontainer -v $(pwd):/backup ubuntu bash -c "cd /dbdata && tar xvf /backup/backup.tar --strip 1"

17.8.3. File System Sharing

  • Before, osxfs~/~gRPC-FUSE + Hypervisor Framework was used on Mac, now VirtioFS + Virtualization Framework is used

17.9. Hosts file

  • Cannot modify/override /etc/hosts file when building an image
  • Ways to modify /etc/hosts
    • Change when running the image
      • Use docker-compose
      • Use docker run
    • Go into the running container and manually modify the hosts file

17.10. Dockerfile

17.10.1. FROM

  • There're 3 ways to specify the target platform. Choose one that you like. For example, building an image on ARM64 and make it AMD64
    • Set a global environment on docker host for building a single image using docker:build and Dockerfile
      • export DOCKER_DEFAULT_PLATFORM=linux/amd64
    • In Dockerfile
    • In docker-compose.yml where image is built docker:compose:platform
  • When an image is run, may encounter this warning
    • WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
    • refer to docker:run:platform for a warning
# x86 version
FROM --platform=linux/x86_64 ubuntu:18.04

17.10.2. CMD, RUN, ENTRYPOINT

17.10.2.1. CMD
  • CMD is to provide defaults for a container that is already created from an image and running.
    • CMD is just to provide paremeters to ENTRYPOINT when docker run is run
      docker run ...longoptions $image $other_command
      $other_command overrides CMD!
    • There can only be one CMD in a Dockerfile or a chain of Dockerfiles included as in parent images
    • See 4
    • If there're multiple, only the last one will run
  • CMD doesn't execute anything at build time and it's not committed to the image.
  • ENTRYPOINT Docker's default entrypoint is to start a shell /bin/sh -c
    • Docker allows you to specify a command in docker run, and that's to combine with entrypoint to run
      • e.g. docker run -it ubuntu <my command> will run /bin/sh -c <my command>
      • where <my command> could be bash
    • Combine ENTRYPOINT, docker run without extra command and CMD
      • ENTRYPOINT ["redis", "-H", "something", "-u", "toto"]
      • CMD ["get", "key"]
      • docker run redisimg is eq. to docker run redisimg get key
  • Specifying entrypoint in docker-compose.yml overrides dockerfile's ENTRYPOINT and clears out dockerfile's CMD

There're 3 forms of CMD 

# exec form, json array, doesn't have variable substitution
CMD ["executable", "param1", "param2"]

# But you can be clever, this works
CMD [ "sh", "-c", "echo $HOME" ]

# as default parameters to ENTRYPOINT
CMD ["param1","param2"]

# shell form
CMD echo $HOME
17.10.2.2. RUN
17.10.2.3. SHELL
  • Change default shell
    • can be multiple SHELL directives
  • SHELL ["executable", "parameters"]
  • ["/bin/sh", "-c"]
    Ubuntu is
    ["/bin/bash", "-c"]
  • ["cmd", "/S", "/C"]
17.10.2.4. /bin/bash vs /bin/sh
  • Most of the time bash is an advanced shell that is on top of sh
  • Sometimes sh is linked to bash
  • Alpine doesn't have /bin/bash

17.10.3. ADD vs COPY

COPY <src>… <dest> COPY ["<src>",… "<dest>"] (this form is required for paths containing whitespace)

ADD and COPY are the same except

  • ADD allows src to be an URL
  • If the src is an archive (tar) it will be unpacked

Always use COPY if it can do what you want

17.10.4. Timezone

  • Works for Ubuntu 

    ENV TZ=America/Toronto
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

    # /etc/localtime -> /usr/share/zoneinfo/Etc/UTC
ll /etc/localtime

cat /etc/timezone
Etc/UTC
  • See MySQL Docker image environment variables

17.10.5. ARG Debian (Jessie)

Problem
When building an image using Debian, a lot of these:
  • debconf: delaying package configuration, since apt-utils is not installed
(no term)
It's like ENV but ENV always overrides ARG
(no term)
Define in Dockerfile without a value
  • So that users can pass at build-time to the builder with the docker build command

  • --build-arg <varname>=<value> 

    FROM busybox
ARG user1
ARG buildno
(no term)

Define ARG after FROM 

ARG VERSION=latest
FROM busybox:$VERSION
ARG VERSION
RUN echo $VERSION > image_version

ARG defined before FROM is outside of build stage so it can't be used in any other instruction

(no term)

Define with a value 

ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y apt-utils

# eq. to
RUN DEBIAN_FRONTEND=noninteractive apt-get ...
17.10.5.1. Predefined ARG variables

Docker has a set of predefined ARG variables without default values. Use it like without defining ARG in Dockerfile 

docker build -t mycontainer . --build-arg <varname>=<value>
--build-arg HTTP_PROXY=http://user:pass@proxy.lon.example.com

These predefined ARG variables are not saved in docker history so it's good to use them for passing sensitive info HTTP_PROXY http_proxy HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy NO_PROXY no_proxy

17.11. Docker Compose

17.11.1. Basics

docker-compose --version
docker-compose up -d

docker-compose up -d --force-recreate
# Force to recreate containers

docker-compose build --no-cache
# Force to rebuild image for services defined in docker-compose.yml with build directive, and then you need to docker-compose up later

# Use a different docker-compose file
docker-compose -f docker-compose.prod.yml up -d

docker-compose down -v
# remove containers and volumes

docker-compose down ---rmi local
# remove only images that don't have a custom tag set by the `image` field

docker-compose down --rmi all
# remove all images used by any service

docker-compose build
# build new image if there's any build directive

docker-compose up -d
# doesn't build image, it will use existing image

docker-compose up -d --build
# build new image and run

docker-compose ps
# shows containers info started by docker compose in current folder

17.11.2. Docker Compose Ubuntu Installation docker:docker compose:ubuntu

Same for 16.04 and 18.04 Grab latest docker-compose version say it's 1.16.1 

sudo curl -o /usr/local/bin/docker-compose -L "https://github.com/docker/compose/releases/download/1.16.1/docker-compose-$(uname -s)-$(uname -m)"

# add execute permission
sudo chmod +x /usr/local/bin/docker-compose
docker-compose -v

# Test
nano docker-compose.yml
my-test:
  image: hello-world

# run
docker-compose up
docker rm <container-id>
docker rmi hello-world

17.11.3. YML

17.11.3.1. version (top level)
Compose and Docker Engine compatibility matrix
https://docs.docker.com/compose/compose-file/
Version 3 Doc
https://docs.docker.com/compose/compose-file/compose-file-v3/
version: '2'
17.11.3.2. environment 
db:
  image: mysql:5.7.40
  environment:
    - MYSQL_ROOT_PASSWORD=example
17.11.3.3. volumes
  1. Top Level
    version: "3.9"
services:
  web:
    image: nginx:alpine
    volumes:
      # Long syntax. Volume #1
      # https://docs.docker.com/compose/compose-file/compose-file-v3/#volume-configuration-reference
      - type: volume
        # first encounter of named volume `mydata`, and it's initialized
        # because `mydata` will be reused in other service `db`, so a top-level volume is needed
        source: mydata
        target: /data
        volume:
          # flag to disable copying of data from a container when a volume is created
          nocopy: true
      # Long syntax. Volume #2
      - type: bind
        source: ./static
        target: /opt/app/static
      # Short syntax. Volume #3. data-volume needs to be at top-level
      - data-volume:/var/lib/mysql
  db:
    image: postgres:latest
    volumes:
      - "/var/run/postgres/postgres.sock:/var/run/postgres/postgres.sock"
      - "dbdata:/var/lib/postgresql/data"
# Top Level
volumes:
  # Named volume which can be used in all services
  # this volume isn't mapped to anything in docker host!
  # top-level volumes can't specify path to named volume.. You have to install other Docker plugin
  mydata:
  dbdata:
  data-volume:
  2. Short syntax
    Modes
    https://docker-docs.netlify.app/docker-for-mac/osxfs-caching/#tuning-with-consistent-cached-and-delegated-configurations
    • Has no effect on old gRPC-FUSE e.g. (osxfs)
    # Syntax [SOURCE:]TARGET[:MODE]
# modes (comma separated)
#   - ro (read only), rw (read and write)
#   - consistent (default) :: perfect consistency (host and container have an identical view of the mount at all times)
#   - delegated :: the container’s view is authoritative (permit delays before updates on the container appear in the host). Writes performed by containers may not be immediately reflected on the host file system. In situations such as NFS asynchronous mode, if a running container with a delegated bind mount crashes, then writes may be lost.
#   - cached :: the host’s view is authoritative (permit delays before updates on the host appear in the container). Writes performed by containers are immediately visible to the host, but there may be a delay before writes performed on the host are visible within containers.
volumes:
  # Just specify a path and let the Engine create a volume
  - /var/lib/mysql

  # Specify an absolute path mapping
  - /opt/data:/var/lib/mysql

  # Path on the host, relative to the Compose file
  - ./cache:/tmp/cache

  # User-relative path
  - ~/configs:/etc/configs/:ro

  # Named volume
  - datavolume:/var/lib/mysql
17.11.3.4. networks (top level) links (deprecated)

Every time docker-compose up creates a single network (per app) which all services in the app are on. Let's say yml file is a folder called foldername This avoids the need to use links 

networks:
;; # network name is foldername_app_net
  app_net:
;; # bridge on a single host (default) and overlay on a Swarm
    driver: bridge
    ipam:
      driver: default
      config:
      - subnet: 172.18.0.0/16
        gateway: 172.18.0.1

Then in a service, specify the ip 

services:
 db:
   image: mysql:5.7.9
   container_name: db
   hostname: db
;; # links is deprecated. 
;; # links:
;; #   - thisserviceorotherservicename:mysql
;; # Use networks:aliases below for thisservice
   networks:
     app_net:
       ipv4_address: 172.18.0.2
       aliases:
         - mysql
   restart: always
  1. Use network defined in another docker-compose
    ;; # ./app1/docker-compose.yml
networks:
  app_net:
    driver: bridge
    ipam:
      driver: default
      config:
      - subnet: 172.16.2.0/24
        gateway: 172.16.2.1
services:
  lucee:
    ...
    networks:
      app_net:
        ipv4_address: 172.16.2.3

;; # ./app2/docker-compose.yml
networks:
  app1_app_net:
    external: true
services:
  nodejs:
    ...
    networks:
      app1_app_net:
        ipv4_address: 172.16.2.4
17.11.3.5. services
  1. Basics
    services:
  lucee:
    build: .
    # use Dockerfile to build an image
    image: cflucee:latest
    # specify image name which can be later used in other service
    # When there's build in yml, you need to add --build
    # docker-compose up -d --build
    container_name: lucee
    ports:
      - "80:80"
      - "8888:8888"
    volumes:
      - /host/path/to/folder1:/root/db/
      - /host/path/to/folder2:/var/log/nginx
    # equivalent to -t in docker run
    tty: true
    networks:
      app_net:
        ipv4_address: 172.18.0.3
  2. ports

    EXPOSE in Dockerfile and -p in docker run -p 8080

    • Neither specify EXPOSE nor -p
      • service in container will not be accessible from anywhere except from the container itself
    • Only specify EXPOSE
      • The service in the container is not accessible from outside Docker, but from inside other Docker containers.
      • Good for inter-container communication
    • Specify EXPOSE and -p
      • The service in the container is accessible from anywhere, even outside Docker
    • Specify -p but not EXPOSE
      • It's like only specify EXPOSE
  3. user 
    clickhouse:
  image: yandex/clickhouse-server
  ports:
    - "8123:8123"
  user: "101:101"
17.11.3.6. Resources

compose file v2 

cpu_count: 2
cpu_percent: 50
cpus: 0.5
cpu_shares: 73
cpu_quota: 50000
cpuset: 0,1

user: postgresql
working_dir: /code

domainname: foo.com
hostname: foo
ipc: host
mac_address: 02:42:ac:11:65:43

mem_limit: 536870912 #512 MB 512*1024*1024, 128MB 134217728
memswap_limit: 2000000000
mem_reservation: 512m
privileged: true

oom_score_adj: 500
oom_kill_disable: true

read_only: true
shm_size: 64M
stdin_open: true
tty: true

compose file v3 

version: '3'
services:
  redis:
    image: redis:alpine
    deploy:
  resources:
    limits:
      cpus: '0.50'
      memory: 50M
      reservations:
        cpus: '0.25'
        memory: 20M
17.11.3.7. Pass arguments from Compose to Dockerfile

docker-compose.yml 

version: '2'
services:
  web:
    build:
      context: ./web
      args:
        REQUIREMENTS: "envs/dev.env"

Dockerfile 

FROM python:3.5
ENV PYTHONUNBUFFERED 1
ENV APP_ROOT /usr/src/app
ARG REQUIREMENTS
...
COPY $REQUIREMENTS $APP_ROOT/
17.11.3.8. platform 
services:  
  frontend:  
    platform: linux/amd64
    build: frontend  
    ports:
  - 80:80
    depends_on:
  - backend
  backend:  
    platform: linux/amd64
    build: backend

17.12. Docker for Windows

Settings > General > Expose daemon on tcp://localhost:2375 without TLS. This is to allow legacy clients/applications such as phpStorm to connect to the daemon.

17.12.1. PowerShell Auto Complete

Win + x > PowerShell (Admin) Run these 

;; # To allow downloaded scripts signed by trusted publishers to run on this computer
Set-ExecutionPolicy RemoteSigned

;; # Type y, check if this return RemoteSigned
get-executionpolicy

;; # Install PowerShell module posh-docker, it may also install NuGet package manager
Install-Module posh-docker

;; # Enable autocompletion for the current PowerShell only
Import-Module post-docker

;; # To make it persistent across all PowerShell sessions, we need to import the module to ~$PROFILE~
;; # Run this directly on PowerShell to create a profile if not exists and add the line
if (-Not (Test-Path $PROFILE)) {
    New-Item $PROFILE –Type File –Force
}

Add-Content $PROFILE "`nImport-Module posh-docker"

;; # check
cat $PROFILE

17.13. Docker Cloud

Docker Cloud uses Docker Hub as native registry.

Use Docker Hub credentials to docker login

Assume your Docker Hub username is abc, give a tag for your image docker tag myimage:mytag abc/myimage:mytag

docker push abc/myimage:mytag

docker logout

17.14. Docker registry

# docker login [OPTIONS] [SERVER]
# Options
# - --password, -p
# - --password-stdin
# - --isername, -u
docker login my.dockerregistry.com
docker login localhost:8080

# push a local image to registry
# First, tag the local image with the host name or IP address, and the port of the registry
docker tag myimage:latest my.dockerregistry.com/myimage:latest
docker push my.dockerregistry.com/myimage:latest

17.15. Common DevOps

17.15.1. Files are in container

container path to code and files dcontainerpath :: /var/www/html store code and files to this folder :: html

17.15.1.1. Makefile at remote Prod/Live host /root/Makefile
Prepare in container a script to dump db
/root/dump.sh
#!/bin/bash
mysqldump -u root --password='$mypass$' livedbname > /root/myproject.sql

.PHONY: info rmbkfolder bkfolder bkdb bkdbsamename tarall tarallexcludesamename drytarallexcludesamename tarcode taruploads taruploadsexclude

filename := $(shell date +%F)
# WordPress runs in container
dcontainer = mycontainername

# website root folder
dcontainerpath = /var/www/html

# container db dump script
dconainterdbdump = /root/dump.sh

# db dump location in container
dcontainerdbdumppath = /root

# host backup folder /root/html
bkpath = html

tarexclude = --exclude='wp-content/uploads' --exclude='sql-admin'
taruploadsexclude = --exclude='backupbuddy_backups'
tarallexclude = --exclude='wp-content/uploads/backupbuddy_bakcups' --exclude='sql-admin'

# myproject.sql
projectname = myproject

# prepare files to push to Live environment
transferpath = /web-root/transfer-files/dev-to-prod/source/

info:
    @echo first run bkfolder
    @echo then run make tarcode and make taruploads
    @echo run make tarall
    @echo run make taruploadsexclude if some directories need to be excluded
    @echo Finally, run rmbkfolder to release space

bkfolder: rmbkfolder
    @echo should use docker cp -a if it is supported
    docker cp -L $(dcontainer):$(dcontainerpath) $(bkpath)

rmbkfolder:
    rm -rf $(bkpath)

bkdb:
    docker exec $(dcontainer) $dcontainerdbdump
    docker cp $(dcontainer):$(dcontainerdbdumppath)/$(projectname).sql $(projectname)-$(filename).sql

bkdbsamename:
    docker exec $(dcontainer) $dcontainerdbdump
    docker cp $(dcontainer):$(dcontainerdbdumppath)/$(projectname).sql $(projectname).sql

tarall:
    tar -czpf all-$(filename).tar.gz -C $(bkpath) .

tarallexcludesamename:
    tar -czpf all.tar.gz -C $(bkpath) . $(tarallexclude))

drytarallexcludesamename:
    tar -cvzpf - -C $(bkpath) . $(tarallexclude)

tarcode:
    tar -czpf code-$(filename).tar.gz -C $(bkpath) . $(tarexclude)

taruploads:
    tar -czpf uploads-$(filename).tar.gz -C $(bkpath)/wp-content/uploads .

taruploadsexclude:
    tar -czpf uploads-exclude-$(filename).tar.gz -C $(bkpath)/wp-content/uploads . $(taruploadsexclude)

# copy code.tar.gz to /myproject/code.tar.gz
# cd /myproject && tar -xvzf code.tar.gz
# make wp-content/uploads directory
# mkdir wp-content/uploads && tar -xvzf uploads.tar.gz -C wp-content/uploads

cleantars:
    rm -rf all.tar.gz

# copy dev code to a transfer folder, remove wp-config.php and .htaccess

pushcodetolive:
    rsync -av ./$(bkpath)/ $(transferpath)
    rm $(transferpath)wp-config.php
    rm $(transferpath).htaccess

Also prepare on Live host /root/devops-db.sh 

#!/bin/bash
cd /root && make bkdbsamename

And /root/devops-files.sh 

#!/bin/bash
cd /root && make bkfolder && make tarallexcludesamename
17.15.1.2. Sql dump inside container

Inside container /var/transfer/bash/dbdump.sh 

#!/bin/bash

# Move to MySQL Dump Folder :
cd /var/transfer/sql-dump/

# Remove dump folder (if Exists):
rm -rf dev-to-prod

# Create and Enter Dump Folder :
mkdir dev-to-prod
cd dev-to-prod

# MySQL Dump A Copy of the Database to the server :
mysqldump -u root --password='dbpassword' dbname > containername.sql

# Find && Replace Dev URL with Prod URL :
sed 's|http://www\.dev.mywebsite.com\.com|http://www\.mywebsite\.com|g' containername.sql
17.15.1.3. local dev /root/Makefile

Create /root/myfile to store ssh plain password 

echo '$mysshpass$' > myfile

Live environment has IP 1.2.3.4 Dev uses container devcontainer and database name is devdb Dev container map files to dev host at /myproject/html

make importlivedb

/root/Makefile 

.PHONY: importlivedb importlivefiles

liveip=1.2.3.4
liveuser=root
livescriptdb=/root/devops-db.sh
livescriptfiles=/root/devops-files.sh
livedbfile=/root/myproject.sql
livedbfilename=myproject.sql

livetarfile=/root/all.tar.gz

containername=devcontainer

password=$$mydevdbpassEscapeDollarSigns$$
dbname=devdb
htmlpath=/myproject/html
htmlparent=/myproject

apacheuserandgroup=www-data:www-data

importlivedb:
        @echo generate sql dump on Live
        sshpass -f myfile ssh -o StrictHostKeyChecking=no $(liveuser)@$(liveip) '$(livescriptdb)'
        @echo download dump to here
        sshpass -f myfile scp -o StrictHostKeyChecking=no $(liveuser)@$(liveip):$(livedbfile) .
#       ssh host1 'mysqldump -u user --password='"'"'$(dbpw)'"'"' dbname > ~/cs-devops/db.sql'
#       scp host1:~/cs-devops/db.sql db/
        @echo change url in db file before import
        sed -i 's|http://www\.myweb\.com|http://dev\.myweb\.com|g' $(livedbfilename)
        sed -i 's|http://myweb\.com|http://dev\.myweb\.com|g' $(livedbfilename)
        cat $(livedbfilename) | docker exec -i $(containername) /usr/bin/mysql -u root --password='$(password)' $(dbname)

importlivefiles:
        sshpass -f myfile ssh -o StrictHostKeyChecking=no $(liveuser)@$(liveip) '$(livescriptfiles)'
        sshpass -f myfile scp -o StrictHostKeyChecking=no $(liveuser)@$(liveip):$(livetarfile) ./all.tar.gz
        rm -rf html
        mkdir html
        tar -xzf all.tar.gz -C ./html
        cp -a $(htmlpath)/wp-config.php .
        cp -a $(htmlpath)/.htaccess .
        rm -rf $(htmlpath)
        cp -a html $(htmlparent)/
        cp -a wp-config.php $(htmlpath)/
        cp -a .htaccess $(htmlpath)/
        chown -R $(apacheuserandgroup) $(htmlpath)

cleanupimport:
        rm -rf all.tar.gz html $(livedbfilename)

17.15.2. Check container status and restart services

#!/bin/bash
container=mycontainer
containerstatus=$(docker inspect -f {{.State.Running}} $container)

if [ "$containerstatus" != "true" ]; then
  echo "$container container is not started. Restarting.."
  docker restart $container
fi

containerstatus=$(docker inspect -f {{.State.Running}} $container)

if [ "$containerstatus" != "true" ]; then
  echo "By now container should be started but it is not.. Abort restarting mysql and apache in container"
else
  echo "Check mysql status in container"
  docker exec $container service mysql status
  if [ $? = "0" ]; then
    echo "mysql is running"
  else
    echo "Restarting mysql in container.."
    docker exec $container service mysql restart
    docker exec $container service mysql status
    # command exit status, command exit code, 0 is successful
    if [ $? = "0" ]; then
  echo "mysql is restarted"
    else
  echo "mysql could not be restarted."
    fi
  fi

  echo "Check apache in container"
  docker exec $container service apache2 status
  if [ $? = "0" ]; then
     echo "apache is running"
  else
   echo "Restarting apache in container.."
   docker exec $container service apache2 restart
   docker exec $container service apache2 status
   if [ $? = "0" ]; then
     echo "apache is restarted"
   else
     echo "apache could not be restarted."
   fi
  fi

fi

17.15.3. Replace string in .sql

https://github.com/tazotodua/useful-php-scripts/blob/master/database-modifier-when-migrating-wordpress https://interconnectit.com/products/search-and-replace-for-wordpress-databases/ https://en-ca.wordpress.org/plugins/wp-migrate-db/ https://wordpress.org/plugins/better-search-replace/ 

UPDATE wp_options SET option_value = replace(option_value, 'http://olddomain.com', 'http://newdomain.com') WHERE option_name = 'home' OR option_name = 'siteurl';

UPDATE wp_posts SET guid = replace(guid, 'http://olddomain.com','http://newdomain.com');

UPDATE wp_posts SET post_content = replace(post_content, 'http://olddomain.com', 'http://newdomain.com');

UPDATE wp_postmeta SET meta_value = replace(meta_value, 'http://olddomain.com', 'http://newdomain.com');

image:tatemz/wp-cli

17.15.4. Download files

csftp=ftp://example.com/var/www/html/
csftpuser=a
csftpp=b

dfolder1:
        wget --user=$(csftpuser) --password='$(csftpp)' -cr -l inf -nH --cut-dirs=3 -P bk/page/ $(csftp)folder1

dfolder2:
        wget --user=$(csftpuser) --password='$(csftpp)' -cr -l inf -nH --cut-dirs=3 -P bk/page/ $(csftp)folder2

dall: dfolder1 dfolder2

17.16. Kubernetes - K8s

Container
a collection of software processes unified by one namespace, with ac cess to an OS kernal that it shares with other containers and little to no access between containers
  • Not tied to infrastructure - only needs Docker Engine
  • Run as isolated processes in user space on the host OS
(no term)
Virtual Machine
  • one or many applications
  • Necessary binaries and libraries
  • The entire guest OS to interact with the applications
Docker instance
a runtime instance of a Docker image contains:
  • A Docker image
  • An execution environment
  • A standard set of instructions
Docker Engine
runtime and packaging tools that must be installed on the hosts that run Docker
Docker Store or Docker Hub
online cloud service where users can store and share Docker images
Kubernetes
a platform to schedule and run containers on clusters of virtual machines. It runs on bare metal, virutal machines, private datacenter and public cloud
  • Written in Go by Google

  • Master Node

    • Kube API Server (communication)
    • Scheduler (scheduling)
    • Monitor created Pods which do not have a Node design yet and design the Pod to run on a specific Node
      • Cluster Manager (controllers)
    • Background threads that run tasks in a cluster. Carry several roles but compiled into 1 single binary
    • worker state
    • maintain the correct number of Pods. Now it's replaced by deployment and ReplicaSets
      ReplicaSet
      ensure that a specified number of replicas for a pod are running at all times
      deployment
    • join services and Pods together
    • handle access management
      etcd database
      simple distributed key value pairs which is all cluster data
      kubectl app
      job scheduling info, Pod details, stage info. CLI
    • server info and authentication info to access Kube API Server

  • communicate back with Master Node

    kubelet process
    this agent sees if Pods have been designed to the Nodes, execute Pod containers via the container engin, mount and run Pod volume and secrets, is aware of Pod of Node states
    kube-proxy process
    Network Proxy and load balancer for the service, on a single Worker Node. It handles the networking routing for TCP and UDP Packets, performs connection forwarding. It might expose to the public internet
    (no term)
    Docker daemon
    Pods
    a Pod is a group of containers, the smallest unit that can be scheduled as a deployment in K8s
    • shared storage, Linux name space, IP addresses amongst other things
    • pending, running, succeeded, failed (any non-zero exit), CrashLoopBackOff (fail to start
    • containers
(no term)
Docker Swarm
  • Written in Go

17.17. Images

17.17.1. image:alpine 

apk add --no-cache libstdc++ \
  && apk add --no-cache --virtual .build-deps \
     curl \
  && apk del .build-deps

# --virtual .somename means the packages to be installed (curl) and their dependencies are not installed to global packages. Once the virtual package .somename is apk del, the packages specified those dependencies are removed

# Shell access, enter into the container
docker exec -it my_container ash

17.17.2. image:debian

  • image:debian:jessie - Debian 8
    • Parent: scratch (perfect for building base images such as debian and busybox
  • image:debian:stretch - Debian 9

17.17.3. image:buildpack-deps:jessie-scm

Parent: buildpack-deps:jessie-curl Available: git, curl, wget, openssh-client

17.17.4. image:buildpack-deps:stretch-curl

Parent: 2

17.17.5. image:php:5.6.31-apache-jessie image:php:apache

Short name: php:5.6.31-apache Parent: 1 Dockerfile

Refer to apache:config

  • PHP_INI_DIR
    • image:php:/usr/local/etc/php (/conf.d)

APACHE_CONFDIR :: etc/apache2 APACHE_ENVVAR :: /etc/apache2/envvars APACHE_RUN_USER :: www-data APACHE_DOCUMENT_ROOT, WORKDIR :: /var/www/html APACHE_LOG_DIR :: /var/log/apache2

EXPOSE 80 CMD ["apache2-foreground"] /usr/local/bin/apache2-foreground

/var/log/apache2/error.log => /dev/stderr /var/log/apache2/access.log => /dev/stdout /var/log/apache2/other_vhosts_access.log => /dev/stdout

copy host config/php.ini to usr/local/etc/php/php.ini (name has to be exact) or copy php-prod.ini to /usr/local/etc/php/conf.d

docker-php-ext-install uses docker-php-ext-enable which adds docker-php-ext-{ext-name}.ini to usr/local/etc/php/conf.d

If you want to get some from PHP's source, you can extract, use and delete it. 

FROM php:7.0-apache
RUN docker-php-source extract \
    # do important things \
    && docker-php-source delete

Install PHP Core Extensions 

FROM php:7.0-fpm
RUN apt-get update && apt-get install -y \
    libfreetype6-dev \
    libjpeg62-turbo-dev \
    libmcrypt-dev \
    libpng-dev \
    && docker-php-ext-install -j$(nproc) iconv mcrypt \
    && docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ \
    && docker-php-ext-install -j$(nproc) gd

Install PECL Extensions use docker-php-ext-enable 

FROM php:7.1-fpm
RUN pecl install redis-3.1.0 \
    && pecl install xdebug-2.5.0 \
    && docker-php-ext-enable redis xdebug
FROM php:5.6-fpm
RUN apt-get update && apt-get install -y libmemcached-dev zlib1g-dev \
    && pecl install memcached-2.2.0 \
    && docker-php-ext-enable memcached

Install other extensions 

FROM php:5.6-apache
RUN curl -fsSL 'https://xcache.lighttpd.net/pub/Releases/3.2.0/xcache-3.2.0.tar.gz' -o xcache.tar.gz \
    && mkdir -p xcache \
    && tar -xf xcache.tar.gz -C xcache --strip-components=1 \
    && rm xcache.tar.gz \
    && ( \
    cd xcache \
    && phpize \
    && ./configure --enable-xcache \
    && make -j$(nproc) \
    && make install \
    ) \
    && rm -r xcache \
    && docker-php-ext-enable xcache

The docker-php-ext-* scripts can accept an arbitrary path, but it must be absolute (to disambiguate from built-in extension names), so the above example could also be written as the following: 

FROM php:5.6-apache
RUN curl -fsSL 'https://xcache.lighttpd.net/pub/Releases/3.2.0/xcache-3.2.0.tar.gz' -o xcache.tar.gz \
    && mkdir -p /tmp/xcache \
    && tar -xf xcache.tar.gz -C /tmp/xcache --strip-Components=1 \
    && rm xcache.tar.gz \
    && docker-php-ext-configure /tmp/xcache --enable-xcache \
    && docker-php-ext-install /tmp/xcache \
    && rm -r /tmp/xcache

copy host config/php.ini to usr/local/etc/php/php.ini (name has to be exact) or copy php-prod.ini to /usr/local/etc/php/conf.d

17.17.5.1. Sample
FROM php:5.6.31-apache-jessie

# apt-utils has to be installed to avoid throwing errors during installation on Debian
RUN set -ex; \
    \
    apt-get update && apt-get install -y \
        apt-utils \
        nano \
        libfreetype6-dev \
        libmcrypt-dev \
        libjpeg62-turbo-dev \
        libpng-dev \
    && \
    apt-get clean && rm -rf /var/lib/apt/lists/* && \
    \
    docker-php-ext-install -j$(nproc) iconv mcrypt && \
    docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ && \
    docker-php-ext-install -j$(nproc) gd mysqli mysql pdo pdo_mysql pcntl zip opcache

COPY config/php-prod.ini /usr/local/etc/php/conf.d/

RUN { \
        echo 'opcache.memory_consumption=128'; \
        echo 'opcache.interned_strings_buffer=8'; \
        echo 'opcache.max_accelerated_files=4000'; \
        echo 'opcache.revalidate_freq=2'; \
        echo 'opcache.fast_shutdown=1'; \
        echo 'opcache.enable_cli=1'; \
    } > /usr/local/etc/php/conf.d/opcache-recommended.ini

# enable apache modules
RUN a2enmod rewrite expires

VOLUME /var/www/html

# TODO chown -R www-data:www-data /var/www/html

CMD ["apache2-foreground"]

17.17.6. image:php53

17.17.6.1. image:bylexus:apache-php53

Ubuntu 12.04 (LTS), Apache 2.2, PHP 5.3.10 Ubuntu Server 12.04, based on ubuntu docker image apache2 php5 php5-cli libapache2-mod-php5 php5-gd php5-ldap php5-mysql php5-pgsql

docker pull bylexus/apache-php53 https://hub.docker.com/r/bylexus/apache-php53/

document root /var/www

docker run -d -p 8080:80 \ -v /home/user/webroot:/var/www \ -e PHP_ERROR_REPORTING='E_ALL & ~E_STRICT' \ bylexus/apache-php53 -v [local path]:/var/www maps the container's webroot to a local path -p [local port]:80 maps a local port to the container's HTTP port 80 -e PHP_ERROR_REPORTING=[php error_reporting settings] sets the value of error_reporting in the php.ini files.

Apache Logs docker logs -f container-id

  • ErrorLog /dev/stdout
  • CustomLog /dev/stdout combined

/etc/php5/apache2/php.ini /etc/php5/cli/php.ini

Default:

  • display_errors=ON
  • error_reporting=E_ALL & ~E_DEPRECATED & ~E_NOTICE
    • change it via -e PHP_ERROR_REPORTING=…
17.17.6.2. image:orsolin/php:5.3-apache

https://github.com/cristianorsolin/docker-php-5.3-apache

This one is Debian and has curl 

phphoht:
    image: orsolin/docker-php-5.3-apache
    container_name: phphoht
    volumes:
      - "./:/var/www/html"
    networks:
      app_nethoht:
        ipv4_address: 172.19.2.3
    ports:
       - "32003:80"
    depends_on:
      - dbhoht
    links:
      - dbhoht

17.17.7. image:php:7

  • Tag latest is 7.2.3-cli-stretch
  • php:7.2.3-apache image:php:7:apache
  • ftp mysqlnd mbstring ftp password-argon2 sodium curl libedit openssl zlib

17.17.8. image:php:7-fpm

  • Tag latest is php:7.2-fpm
  • FPM is for Nginx
  • PHP_INI_DIR
    • /usr/local/etc/php/php.ini
    • /usr/local/etc/php/conf.d/*.ini
      • Setting of extensions installed by docker-php-ext-enable
    • Restart using docker-compose restart phpcontainername

17.17.9. image:python

  • python:3
    • python:3.8.2-buster

  • Basics 

    # Run Python Interpreter interactively
docker run -it --rm --name mypy python:3

#Map current folder and run hello.py
docker run -it --rm --name mypy -v "$PWD":/usr/src/myapp -w /usr/src/myapp python:3.8.2-buster python hello.py

17.17.10. image:mongo

Last used
mongo:4.2.5-bionic
(no term)
Other related images

17.17.11. image:mysql

  • https://hub.docker.com/_/mysql
  • Parent
    5.7.9
    1
    5.7.40
    image:oraclelinux:7-slim
    Before MySQL v8
    no ARM version
  • mysql:mysql
  • mysqld
  • EXPOSE 3306
  • -v /my/mysql-data:/var/lib/mysql
  • -v /my/mysql-log:/var/log/mysql
  • -v /my/mysql-conf:/etc/mysql/conf.d
    Main my.cnf can be overriden
    -v /my/my.cnf:/etc/mysql/my.cnf
  • if you don't want to throw in my.cnf or conf.d
    • Use 3.1 or 4
    • docker run -it --rm mysql:tag --verbose --help
  • docker exec -it some-mysql bash
  • ARM64
    • Start with arm64v8/mysql:8.0
17.17.11.1. Load configs
Startup config
/etc/mysql/my.cnf and any .cnf files under /etc/mysql/conf.d
(no term)
Mount host directory to /etc/mysql/conf.d in container using -v in docker run
(no term)
Instead of loading custom conf.d directory, tags --tag-name can be used adjust the configs
(no term)
Use 3.1 or docker-compose command
(no term)

For container that couldn't have more than 128M memory, use innodb-buffer-pool-size to scale it down 

docker run -it --rm --name my-mysql-container \
  -e MYSQL_ROOT_PASSWORD=my-secret-pw \
  -d mysql:5.7.9 \
  --character-set-server=utf8mb4 \
  --collation-server=utf8mb4_unicode_ci
  --innodb-buffer-pool-size=50M

In docker-compose.yml 

command:
  mysqld --innodb-buffer-pool-size=50M
(no term)
More MySQL Config
17.17.11.2. Environment vars
  • None of these will have effect if the container is started with a data dir that already contains a db:
    MYSQL_ROOT_PASSWORD
    required
    MYSQL_DATABASE
    opt. a db to be created on image startup
    MYSQL_USER, MYSQL_PASSWORD
    opt. create a user for MYSQL_DATABASE
    (no term)

    These vars can be loaded through a file 

    docker run -it --rm -e MYSQL_ROOT_PASSWORD_FILE=/run/secrets/mysql-root mysql:5.7
  • e.g. TZ=America/Toronto
17.17.11.3. Import .sql on image startup

Poibt a directory that has a .sh, .sql, and .sql.gz to a dir in container 

-v /host/path/to/db-dir:/docker-entrypoint-initdb.d

Importing needs a lot of memory. If host doesn't have 128M memory, try to import it locally and then rsync the persist data folder to host and chown -R 999:docker /path/to/persistdir accordingly on host.

17.17.11.4. Access, view logs, db dump
docker exec -it my-mysql-container bash
docker logs my-mysql-container
docker exec some-mysql sh -c \
  'exec mysqldump --all-databases -uroot -p"$MYSQL_ROOT_PASSWORD"' > /some/path/on/your/host/all-databases.sql
17.17.11.5. Persist data

A place to persist db data on host :: /my/own/datadir 

-v /my/own/datadir:/var/lib/mysql

For dev where db data can be imported and disposed when needed, use volume (e.g. dbdata) 

version: '2'

volumes:
  dbdata:

networks:
  app_net:
    driver: bridge
    ipam:
      driver: default
      config:
      - subnet: 172.18.0.0/16
        gateway: 172.18.0.1

services:
  db:
    image: mysql:5.7.9
    container_name: db
    volumes:
      - "dbdata:/var/lib/mysql"
      - "/my/own/datadir:/docker-entrypoint-initdb.d"
    networks:
      app_net:
        ipv4_address: 172.18.0.2
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: d7
      MYSQL_DATABASE: d7
      MYSQL_USER: d7
      MYSQL_PASSWORD: d7
17.17.11.6. Example

To initialize a db, use docker run with data persist 

docker run -it --name my-db-container \
  -v /host/path/to/dbdata:/var/lib/mysql \
  -v /host/path/to/db-dir:/docker-entrypoint-initdb.d \
  -e MYSQL_ROOT_PASSWORD=d7 \ 
  -e MYSQL_DATABASE=d7 \ 
  -e MYSQL_USER=d7 \ 
  -e MYSQL_PASSWORD=d7 \
  -d mysql:5.7.9 \
--innodb-buffer-pool-size=50M \
--innodb-buffer-pool-chunk-size=50M \
--max-allowed-packet=50M

Check db status, then you docker-compose restart the container without specifying import directive in docker-compose.yml 

mydbcontainer:
  image: mysql:5.7.9
  container_name: mydbcontainer
  volumes:
    - "/host/path/to/dbdata:/var/lib/mysql"
  command:
    mysqld --innodb-buffer-pool-size=50M --innodb-buffer-pool-chunk=50M --max-allowed-packet=16M
  restart: always
  networks:
    ...

version: '3.5'

services:
  mydb:
    container_name: mydb
    image: mysql:5.7.40
    platform: linux/amd64
    command: --default-authentication-plugin=mysql_native_password
    volumes:
      # Include this line for importing .sql to initialize database mymysqldb
      # - ../../../my-mysql-data:/var/lib/mysql:cached
      - ./setupfiles/mysql/my.cnf:/etc/mysql/my.cnf
      - mydbdata: /var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=myrootpass
      # Include this line for imporiting .sql to initialize database mymysqldb 
      - MYSQL_DATABASE=mymysqldb
    ports:
      - "9006:3306"
    networks:
      - default

17.17.12. image:wordpress image:wordpress:apache

  • wordpress:4.9.3-php7.1-apache
    Parent
    php:7.1-apache image:php:apache image:php:7:apache
  • ghostscript libfreetype6-dev libjpeg-dev libmagickwand-dev libpng-dev libzip-dev
  • php:gd mysqli opcache bcmath exif zip
17.17.12.1. docker-compose.yml
version: '2'
volumes:
  dbsandboxwp:
networks:
  app_netsandboxwp:
    driver: bridge
    ipam:
  driver: default
  config:
  - subnet: 172.20.0.0/24
    gateway: 172.20.0.1
services:
  dbsandboxwp:
    image: mysql:5.7.9
    container_name: dbsandboxwp
    networks:
  app_netsandboxwp:
    ipv4_address: 172.20.0.2
    ports:
  - "31000:3306"
    restart: always
    environment:
  MYSQL_ROOT_PASSWORD: wp
  MYSQL_DATABASE: wp
  MYSQL_USER: wp
  MYSQL_PASSWORD: wp

  phpsandboxwp:
    image: wordpress:4.9.3-php7.1-apache
    container_name: phpsandboxwp
    volumes:
  - "./:/var/www/html"
    restart: always
    networks:
  app_netsandboxwp:
    ipv4_address: 172.20.0.3
    environment:
  WORDPRESS_DB_HOST: dbsandboxwp
  WORDPRESS_DB_PASSWORD: wp
  WORDPRESS_DB_USER: wp
  WORDPRESS_DB_NAME: wp
  WORDPRESS_TABLE_PREFIX: wp_
    # environment variables honor mysql
    # WORDPRESS_DB_HOST WORDPRESS_DB_PASSWORD
    # environment variables default
    # WORDPRESS_DB_USER root
    # WORDPRESS_DB_NAME wordpress
    # WORDPRESS_TABLE_PREFIX ""
    ports:
  - "32000:80"
    depends_on:
  - dbsandboxwp
    links:
  - dbsandboxwp

17.17.13. image:drupal image:drupal:apache

drupal:7.56-apache Parent :: php:7.0-apache

17.17.13.1. docker-compose.yml

docker run -ti –rm –volumes-from phpsandboxd7 –name html-data ubuntu:xenial docker cp html-data:/var/www/html . 

version: '2'
volumes:
  volsandboxd7:
networks:
  app_netsandboxd7:
    driver: bridge
    ipam:
  driver: default
  config:
  - subnet: 172.20.1.0/24
    gateway: 172.20.1.1
services:
  dbsandboxd7:
    image: mysql:5.7.9
    container_name: dbsandboxd7
    networks:
  app_netsandboxd7:
    ipv4_address: 172.20.1.2
    aliases:
      - mysql
    ports:
  - "31001:3306"
    restart: always
    environment:
  MYSQL_ROOT_PASSWORD: d7
  MYSQL_DATABASE: d7
  MYSQL_USER: d7
  MYSQL_PASSWORD: d7

  phpsandboxd7:
    image: drupal:7.56-apache
    container_name: phpsandboxd7
    volumes:
  - volsandboxd7:/var/www/html
#      - /var/www/html/modules
#      - /var/www/html/profiles
#      - /var/www/html/sites
#      - /var/www/html/themes
    restart: always
    networks:
  app_netsandboxd7:
    ipv4_address: 172.20.1.3
    aliases:
      - drupal
    ports:
  - "32001:80"
    depends_on:
  - dbsandboxd7

17.17.14. image:tatemz/wp-cli image:tatemz/wp-cli

https://github.com/tatemz/docker-wpcli

Refer to wp-cli:search-replace 

docker-compose run --rm my-wpcli
docker-compose run --rm my-wpcli db check
docker-compose run --rm my-wpcli post list
docker-compose run --rm my-wpcli db search 'https?://' --regex --stats

# For some reason, the following only works when run in PowerShell
# This replaces A with B for the current database without changing it but instead export it.
docker-compose run --rm my-wpcli search-replace 'http://live.ca' 'http://localhost:9999' --export=/var/www/html/devops/db/export.sql

# There's no way to search/replace multiple strings. Do it one by one
docker-compose run --rm my-wpcli search-replace 'https://live.ca' 'http://localhost:9999'

# at the end export it
docker-compose run --rm my-wpcli db export --add-drop-table /var/www/html/devops/db/dump.sql

17.17.15. image:wordpress:cli image:wordpress:cli

wordpress:cli-php5.6 wordpress-cli-2-php5.6 wordpress:cli-2.0-php5.6 wordpress:cli-2.0.0-php5.6 wordpress:cli-php7.0 wordpress:cli-php7.1 wordpress:cli-php7.2 

docker pull wordpress:cli-php7.0

docker run -it --rm --volumes-from mywp --network container:mywp wordpress:cli-php7.0 user list

17.17.16. image:lucee docker:image:lucee

Image name
lucee/lucee4-nginx
Tags
https://hub.docker.com/r/lucee/lucee4-nginx/tags
Choose a tag
https://github.com/lucee/lucee-dockerfiles
Parent Images
lucee/lucee4 > tomcat:8.0.36-jre8 > openjdk:8-jre > buildpack-deps:stretch-curl
17.17.16.1. docker-compose.yml Dockerfile

docker-compose.yml 

version: '2'
networks:
  app_net:
    driver: bridge
    ipam:
  driver: default
  config:
  - subnet: 172.16.2.0/24
    gateway: 172.16.2.1
services:
  lucee:
    build: .
    image: cflucee:latest
    container_name: lucee
    ports:
  - "80:80"
  - "8888:8888"
    volumes:
  - /home/you/db/access:/root/db/
  - /home/you/www:/var/www
  # Scheduled tasks are setup in http://127.0.0.1:8888/lucee/admin/web.cfm
  # /home/you/www/WEB-INF/lucee/scheduler/scheduler.xml > /var/wwww/WEB-INF/lucee/scheduler/scheduler.xml
  - /home/you/logs/tomcat:/usr/local/tomcat/logs
  - /home/you/logs/nginx:/var/log/nginx
  - /home/you/logs/lucee:/opt/lucee/web/logs
    networks:
  app_net:
    ipv4_address: 172.16.2.3

Dockerfile

Change it based on the base Dockerfile
https://github.com/lucee/lucee-dockerfiles/blob/master/4.5/Dockerfile
Image has 2 ports exposed
8080 (Tomcat listens but not used) 8888 (used for Lucee Server Admin/Web)
(no term)
Config folders
Tomcat
/usr/local/tomcat/conf
Lucee config for default site
/opt/lucee/web
server.xml
https://github.com/lucee/lucee-dockerfiles/blob/master/4.5/server.xml
web.xml
https://github.com/lucee/lucee-dockerfiles/blob/master/4.5/web.xml
catalina.properties
https://github.com/lucee/lucee-dockerfiles/blob/master/4.5/catalina.properties
Lucee server context
/opt/lucee/server/lucee-server/context
lucee-server.xml
https://github.com/lucee/lucee-dockerfiles/blob/master/4.5/lucee-server.xml
Lucee web
/opt/lucee/web
lucee-web.xml.cfm
https://github.com/lucee/lucee-dockerfiles/blob/master/4.5/lucee-web.xml.cfm
(no term)
Log folders
Tomcat
/usr/local/tomcat/logs
Lucee logs for default site
/opt/lucee/web/logs
(no term)
Environment varialbes 
FROM lucee/lucee4-nginx:4.5.1.024

ENV TZ=America/Toronto
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

# Copy UcanAccess
COPY ucanaccess/* /usr/local/tomcat/lib/

# Custom tags are manually copied to each host's root folder...
COPY config/customtags/* /opt/lucee/web/customtags/
# COPY config/customtags/* /opt/lucee/server/lucee-server/context/library/

# Tomcat configs
# COPY catalina.properties server.xml web.xml /usr/local/tomcat/conf/
COPY config/tomcat/server.xml /usr/local/tomcat/conf/
COPY config/tomcat/context.xml /usr/local/tomcat/conf/
# COPY config/tomcat/legacy_js.xml /usr/local/tomcat/conf/Catalina/127.0.0.1/legacy_js.xml

# Custom setenv.sh to load Lucee
COPY setenv.sh /usr/local/tomcat/bin/
RUN chmod a+x /usr/local/tomcat/bin/setenv.sh

# Default setenv.sh https://github.com/lucee/lucee-dockerfiles/blob/master/4.5/setenv.sh
# Added UcanAccess

# NGINX configs
COPY config/nginx/ /etc/nginx/
# /etc/nginx/nginx.conf :: Default not modified https://github.com/lucee/lucee-dockerfiles/blob/master/lucee-nginx/4.5/nginx.conf
# /etc nginx/default.conf :: https://github.com/lucee/lucee-dockerfiles/blob/master/lucee-nginx/4.5/default.conf

# Add datasources. Default https://github.com/lucee/lucee-dockerfiles/blob/master/4.5/lucee-server.xml
COPY config/lucee/lucee-server.xml /opt/lucee/server/lucee-server/context/lucee-server.xml

# Lucee server PRODUCTION configs
# COPY config/lucee/lucee-web.xml.cfm /opt/lucee/web/lucee-web.xml.cfm

# Deploy codebase to container
# COPY index.cfm /var/www
17.17.16.2. Add ucanaccess db driver to TomCat, Add M$ Access Datasources
  • Refer to docker:image:tomcat
  • Add Access datasource on yourdomain.com:8888/lucee/admin/server.cfm
  • Other - JDBC Driver
  • abc
  • net.ucanaccess.jdbc.UcanaccessDriver
  • jdbc:ucanaccess:///home/MyName/www/database/MyMsAccessDB.mdb

Create a datasource using UI. One of the Lucee setting files will have this line in <datasources> 

<data-source allow="511" 
         blob="false" 
         class="net.ucanaccess.jdbc.UcanaccessDriver" 
         clob="false" connectionTimeout="1" custom="" database="" dbdriver="Other" 
         dsn="jdbc:ucanaccess:///home/MyName/www/database/MyMsAccessDB.mdb" 
         host="" metaCacheTimeout="60000" name="abc" 
         password="abc test password" 
         storage="false" validate="false"/>
17.17.16.3. Lucee server and web settings
Lucee server setting
/opt/lucee/server/lucee-server/context/lucee-server.xml
Individual website setting
/opt/lucee/web/lucee-web.xml.cfm
17.17.16.4. Custom tags (.cfm|.cfc) under /opt/lucee/web/customtags/
17.17.16.5. Change Lucee Server Admin UI password

Use this with a salt to generate hspw in Lucee server setting <cfLuceeConfiguration hspw="" salt="" version="4.5"> Refer to image:lucee:salt. It seems like you should use the salt that is in the default lucee-server.xml. Can't use a random salt.

17.17.16.6. How does it work with Nginx?

Lucee operates on TomCat port 8888 using 127.0.0.1. Nginx forwards 80 incoming traffic to port 8888. By default, only *.cfm|cfc|cfr files are forwarded to 8888. Add more files to /etc/nginx/conf.d/default.conf 

location ~* \.(cfm|cfc|cfr|jpe?g|js|css|png|gif|html)$ {
    proxy_pass http://127.0.0.1:8888;
    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_connect_timeout 600;
    proxy_send_timeout 600;
    proxy_read_timeout 600;
    send_timeout 600;
}
17.17.16.7. Virtual Directory and TomCat

Virtual Directory. Setup docker:image:tomcat:vd first, then setup forwarding 

location ~* /legacy_js {
    proxy_pass http://127.0.0.1:8888;
    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_connect_timeout 600;
    proxy_send_timeout 600;
    proxy_read_timeout 600;
    send_timeout 600;
}
17.17.16.8. Add hosts
server {
    listen 80 default_server;
    server_name domain1.com;
    index  index.cfm index.html index.htm;
    root   /var/www/site1;
    server_name_in_redirect off;

    include allcf/common.conf;

    location ~* (/legacy_js)|(\.(cfm|cfc|cfr|jpe?g|js|gif|css|png|html)$) {
    proxy_pass http://127.0.0.2:8888;
    proxy_redirect off;
    proxy_set_header Host 127.0.0.2;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_connect_timeout 600;
    proxy_send_timeout 600;
    proxy_read_timeout 600;
    send_timeout 600;
    }

}

server {
    listen 80;
    server_name domain2.com;
    index  index.cfm index.html index.htm;
    root   /var/www/site2;
    server_name_in_redirect off;

    include allcf/common.conf;

    location ~* (/legacy_js)|(\.(cfm|cfc|cfr|jpe?g|js|gif|css|png|html)$) {
    proxy_pass http://127.0.0.3:8888;
    proxy_redirect off;
    proxy_set_header Host 127.0.0.3;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_connect_timeout 600;
    proxy_send_timeout 600;
    proxy_read_timeout 600;
    send_timeout 600;
    }

}

Add hosts in docker:image:tomcat:host

17.17.16.9. Scheduled Tasks

Create them in /lucee/admin/web.cfm Checkout this file /var/www/WEB-INF/lucee/scheduler/scheduler.xml

17.17.17. image:tomcat docker:image:tomcat

Image name: tomcat:8.0.36-jre8

17.17.17.1. Settings

The default webapps (including TomCat Admin Console, localhost:8080/manager/html and .jar files) is in /usr/local/tomcat/webapps The default connector port is 8080.

docker:image:lucee deletes this webapps folder, place it under /usr/local/tomcat/lucee which contains all .jar files.

Include that directory in catelina.properties in common.loader=${catalina.home}/lucee/*.jar

Define a <web-app> with multiple servlets in ${catalina.home}/web.xml to handle all .html and .cf(m|c) files.

When the web-app and Lucee servlet are built, new Lucee admin paths (context) are built :: 127.0.0.1:8888/lucee/admin/server.cfm and web.cfm Corresponding paths are /opt/lucee/server(/lucee-server) and opt/lucee/web

In ${catalina.home}/server.xml, define Host(s) as usual. Each <Host> has appBase="webapps" which is a folder under /usr/local/tomcat. Using the empty webapps folder is fine.

Each <Host> can have multiple <Context> with different path/docBase. The root directory of each Host can have:

META-INF folder
context.xml (server-dependent config, e.g. datasource),
WEB-INF folder
web.xml (server-independent config, e.g. servlet mappings, docker:image:lucee:tasks)

If there's ${catalina.home}/context.xml, all hosts have these settings and override <Context> set in <Host> in server.xml and override <Context> in any META-INF/context.xml under all hosts: ${catalina_home}/[enginename]/[host]/META-INF/context.xml

The only time path attribute in <Context> has any effect is in the server.xml

17.17.17.2. Install UcanAccess JDBC driver to enable M$ Access datasource

Copy these jar files 

COPY ucanaccess/* /usr/local/tomcat/lib/

ucanaccess-4.0.1.jar lib/*.jar (commons-lang, commons-logging, hsqldb, jackcess)

Add these 2 lines to setenv.sh 

UCANACCESS_HOME=/usr/local/tomcat/lib;
export UCANACCESS_HOME;

docker:image:lucee 

COPY setenv.sh /usr/local/tomcat/bin/
RUN chmod a+x /usr/local/tomcat/bin/setenv.sh
17.17.17.3. Setup Virtual Directory per Host docker:image:tomcat:vd

In /usr/local/tomcat/conf/server.xml, you can see how many hosts are defined. e.g. 

<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="..." />

  <Service name="Catalina">
    <Connector executor="tomcatThreadPool"
               port="8888" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />

    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

    <Engine name="Catalina" defaultHost="127.0.0.1">
      <Host name="127.0.0.1"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">
        <Valve className="org.apache.catalina.valves.RemoteIpValve"
               remoteIpHeader="X-Forwarded-For"
               requestAttributesEnabled="true" />
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log" suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />

        <Context path="" docBase="/var/www/">
          <JarScanner scanClassPath="false"/>
        </Context>
        <!-- You can define virtual directory here or follow the way of adding vd.xml-->
        <!-- 
        <Context path="/legacy_js" docBase="/path/outside/var/www/legacy_js"></Context>
        -->
      </Host>
    </Engine>

  </Service>

</Server>

Method 1 is to add a <Context> tag in a <Host> in server.xml. But for me it doesn't work when multiple hosts need to have the same vd and it creates a lot of other problems.

General rule :: don't include more than 1 <Context> in each <Host> in server.xml

Method 2 is to create a file legacy_js.xml under ${catalina_home}/conf/Catalina/127.0.0.1/ Where Catalina is the engine name and 127.0.0.1 is the Host name. #+NAME legacy_js.xml 

<?xml version='1.0' encoding='utf-8'?>
<Context docBase="/root/vd/legacy_js/"></Context>

http://127.0.0.1/legacy_js/abc.js points to /root/vd/legacy_js/abc.js

If it's foo#bar.xml then http://127.0.0.1/foo/bar/abc.js

Method 2 is not tested for multiple hosts usage…

Method 3 is tested.

Create relative symbolic link for docker usage. /var/www/site2/legacy_js point to ../site1/source_legacy_js 

cd /var/www/site2
ln -s ../site1/source_legacy_js/ legacy_js

TomCat can't follow relative symbolic link.. Add this line to ${catalina_home}/context.xml for TomCat 8 

<Resources allowLinking="true"></Resources>
;; # For TomCat 7, add attribute allowLinking="true" to the only <Context>

I recently found out Method 3 doesn't work.. Here's Method 4. Don't reverse proxy to TomCat at all, if vd has only static files. In nginx default.conf 

location ~* ^/legacy_js {
;; # nginx serve the static files
}
location ~* \.(cfm|cfc|cfr|jpe?g|js)
17.17.17.4. Add host docker:image:tomcat:host

Host name should be 127/8 ip address otherwise you need to setup DNS. Refer to docker:image:tomcat:log 

<Host name="127.0.0.2" appBase="webapps">
  <Valve className="org.apache.catalina.valves.RemoteIpValve"
         remoteIpHeader="X-Forwarded-For"
         requestAttributesEnabled="true" />
  <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
         prefix="localhost_access_log" suffix=".txt"
         pattern="%h %l %u %t &quot;%r&quot; %s %b" />

  <Context path="" docBase="/var/www/site1/">
    <JarScanner scanClassPath="false"/>
  </Context>
</Host>

<Host name="127.0.0.3" appBase="webapps">
<Valve className="org.apache.catalina.valves.RemoteIpValve"
       remoteIpHeader="X-Forwarded-For"
       requestAttributesEnabled="true" />
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
       prefix="localhost_access_log" suffix=".txt"
       pattern="%h %l %u %t &quot;%r&quot; %s %b" />
<Context path="" docBase="/var/www/site2/">
  <JarScanner scanClassPath="false"/>
</Context>
<Context docBase="/var/www/vd/legacy_js/" path="/legacy_js"></Context>
</Host>
17.17.17.5. Restrict Access

Refer to docker:image:tomcat:log 

<!-- RemoteAddrValue is %a in log -->
<Valve className="org.apache.catalina.valves.RemoteAddrValve"
       allow="allow.com" />
<!-- RemoteHostValue is %h in log -->
<Valve className="org.apache.catalina.valves.RemoteHostValve"
       allow="127.0.0.1,10.100.1.2" />
17.17.17.6. Logging

docker:image:tomcat:log Google :: tomcat access log value pattern

%h
remote host name. Or IP if enableLookups for the connector is false If request is passed from Nginx (proxy_pass), it's 127.0.0.1 If request is directly from Internet, it's the requester's IP
%a
Remote IP address. Same as %h but IP only If request is passed from Nginx (proxy_pass), it's 127.0.0.1 If request is directly from Internet, it's the requester's IP.
%v
Local server name. If request is passed from Nginx (proxy_pass), it's the name of <Host> in Tomcat If request is directly from Internet, it's the request domain name
%A
Local IP address. Tomcat server public IP
%{local}p
local port. Which is 80 or 8888 where is set in <Connector>
%{remote}p
remote port. IDK.. Maybe random
%{xxx}i
incoming header value. e.g. %{X-Forwarded-Host}
%{xxx}o
outgoing header value

17.17.18. image:goaccess docker:image:goaccess

Don't follow the provided instructions..
https://github.com/allinurl/goaccess#docker
(no term)

Follow mine 

git clone https://github.com/allinurl/goaccess.git goaccess && cd $_

docker build . -t allinurl/goaccess

mkdir -p /home/li/goaccess/{data,html,log}

# create config file
nano /home/li/goaccess/data/goaccess.conf

# restart container after config file is changed
docker restart goaccess

docker run --restart=always -d -p 7890:7890 \
  -v "/home/li/goaccess/data:/srv/data"     \
  -v "/home/li/goaccess/html:/srv/report"   \
  -v "/home/li/goaccess/log:/srv/logs"      \
  --name=goaccess allinurl/goaccess         \
  --no-global-config --config-file=/srv/data/goaccess.conf \
       --log-file=/srv/logs/access.log

# entrypoint is `/bin/goaccess`
# default conf file is in the Alpine container :: /etc/goaccess
# so must use --config-file to override
# use --log-file to dynamically point to the log you want to analyze

# generated HTML report is under /srv/goaccess/html/index.html
# open it locally on host machine will have a real-time updated one! Because the webpage uses WebSocket

# completely remove
docker stop goaccess
docker rm goaccess
docker rmi allinurl/goaccess
17.17.18.1. Config goaccess.conf

Better to add the following lines to the default conf or you can just have these lines in the conf 

log-file /srv/logs/access.log
output /srv/report/index.html
real-time-html true

# real-time-html is true is to ensure `docker run -d` runs continuously
# followed by script converted log-format, time-format, etc.
# log-format COMBINED

17.17.19. image:stilliard/pure-ftpd docker:image:ftp

Image name: stilliard/pure-ftpd:hardened Parent image: debian:jessie

https://github.com/stilliard/docker-pure-ftpd By default, it only supports FTP not SFTP

Dockerfile docker build -t myftp . 

FROM stilliard/pure-ftpd:hardened

# e.g. you could change the defult command run:
CMD /run.sh -c 30 -C 5 -l puredb:/etc/pure-ftpd/pureftpd.pdb -E -j -R -P $PUBLICHOST -p 30000:30009

# /usr/sbin/pure-ftpd

# -c 50, --maxclientsnumber (no more than 50 people at once)
# -C 10, --maxclientsperip (no more than 10 requests from the same ip)
# -l puredb:/etc/pure-ftpd/pureftpd.pdb, --login (login file for virtual users)
# -E, --noanonymous (only real users)
# -j, --createhomedir (auto create home directory if it doesn't already exist)
# -R, --nochmod (prevent usage of the CHMOD command)

# -tls 1, Enables optional TLS support

# In Compose

# -P $PUBLICHOST :: setting for PASV support, passed in your the PUBLICHOST env var
# -p 30000:30009 :: PASV port range

docker-compose.yml 

version: '2'
networks:
  app_net:
    driver: bridge
    ipam:
  driver: default
  config:
  - subnet: 172.16.3.0/24
    gateway: 172.16.3.1
services:
  myftp:
    build: .
    image: myftp:latest
    container_name: myftp
    environment:
  - PUBLICHOST=ftp.yourdomain.com
#     - ADDED_FLAGS=-d -d
    ports:
  - "21:21"
  - "30000-30009:30000-30009"
# open these many ports to support Passive Mode..
    volumes:
  - /host/storefiles/ftp:/home/ftpusers
  - /host/config/ftp/passwd:/etc/pure-ftpd/passwd
# persist the user database
# only one file under /etc/pure-ftpd/passwd which is pureftps.passwd and it contains the user database
#     - /host/config/ftp/ssl:/etc/ssl/private
# a single `pure-ftpd.pem` file with server's SSL certificates for TLS support
    networks:
  app_net:
    ipv4_address: 172.16.3.3

Makefile 

FTPCONFIG=/etc/pure-ftpd/passwd/pureftpd.passwd

.PHONY: default info start stop restart list adduser deluser

default:
    @echo "make [command]. Available commands are:"
    @echo "info, start, stop, restart, list, adduser, deluser"

info:
    @echo "List all virutal user: make list"
    @echo "Add a new user abc and make parent folder to /FTP"
    @echo 'make adduser username="abc" folder="FTP"'
    @echo 'Add a new user abc and make parent folder to root /'
    @echo 'make adduser username="abc" folder="application1"'
    @echo 'Delete a user abc: make deluser username="abc"'

start:
    sudo docker-compose up -d --build

stop:
    sudo docker-compose down -v

restart:
    sudo docker-compose down -v && sudo docker-compose up -d --build

enter:
    sudo docker exec -it newcomftp bash -I

list:
    @echo "<account>:<password>:<uid>:<gid>:<gecos>:<home directory>:<upload bandwidth>:<download bandwidth>:<upload ratio>:<download ratio>:<max number of connections>:<files quota>:<size quota>:<authorized local IPs>:<refused local IPs>:<authorized client IPs>:<refused client IPs>:<time restrictions>"
    sudo cat ~/config/ftp/passwd/pureftpd.passwd

adduser:
    sudo docker exec -it myftp pure-pw useradd $$username -f $(FTPCONFIG) -m -u ftpuser -d /home/ftpusers/$$folder

deluser:
    sudo docker exec -it myftp pure-pw userdel $$username -f $(FTPCONFIG) -m

Pureftp Manual

  • Add User
  • FAQ
  • man pure-ftpd
  • Start running /usr/sbin/pure-ftpd $some_flags

ftp:passive mode FTP opens one channel for command and another for data.

Active mode
client establishes the command channel (from client port X to server port 21) but the server establishes the data channel (from server port 20 to client port Y, where Y has been supplied by the client).
Passive mode
client establishes both channels. In that case, the server tells the client which port should be used for the data channel.

17.17.20. image:dockerfile-from-image

image:dockerfile-from-image https://github.com/levonlee/dockerfile-from-image

docker run -v /var/run/docker.sock:/var/run/docker.sock dockerfile-from-image <IMAGE_TAG_OR_ID> e.g. docker run -v /var/run/docker.sock:/var/run/docker.sock dockerfile-from-image ubuntu

For windows docker run -v //var/run/docker.sock:/var/run/docker.sock dockerfile-from-image ubuntu

Dockerfile (added one line: RUN chmod) 

FROM alpine:3.2
MAINTAINER CenturyLink Labs <clt-labs-futuretech@centurylink.com>

RUN apk --update add ruby-dev ca-certificates && \
    gem install --no-rdoc --no-ri docker-api && \
    apk del ruby-dev ca-certificates && \
    apk add ruby ruby-json && \
    rm /var/cache/apk/*

ADD dockerfile-from-image.rb /usr/src/app/dockerfile-from-image.rb
RUN chmod +x /usr/src/app/dockerfile-from-image.rb

ENTRYPOINT ["/usr/src/app/dockerfile-from-image.rb"]
CMD ["--help"]

FROM alpine:3.2
MAINTAINER CenturyLink Labs <clt-labs-futuretech@centurylink.com>

RUN apk --update add ruby-dev ca-certificates && \
    gem install --no-rdoc --no-ri docker-api && \
    apk del ruby-dev ca-certificates && \
    apk add ruby ruby-json && \
    rm /var/cache/apk/*

ADD dockerfile-from-image.rb /usr/src/app/dockerfile-from-image.rb
RUN chmod +x /usr/src/app/dockerfile-from-image.rb

ENTRYPOINT ["/usr/src/app/dockerfile-from-image.rb"]
CMD ["--help"]

dockerfile-from-image.rb (same as GitHub) 

#! /usr/bin/env ruby
require 'docker'
require 'optparse'

NONE_TAG = '<none>:<none>'
NOP_PREFIX = '#(nop) '

options = {}
commands = []

;; # Default to -h if no arguments
ARGV << '-h' if ARGV.empty?

OptionParser.new do |opts|
  opts.banner = "Usage: dockerfile-from-image.rb [options] <image_id>"

  opts.on("-f", "--full-tree", "Generate Dockerfile for all parent layers") do |f|
    options[:full] = f
  end

  opts.on_tail("-h", "--help", "Show this message") do
    puts opts
    exit
  end
end.parse!

image_id = ARGV.pop
abort('Error: Must specify image ID or tag') unless image_id

;; # Collect all image tags into a hash keyed by layer ID.
;; # Used to look-up potential FROM targets.
tags = Docker::Image.all.each_with_object({}) do |image, hsh|
  tag = image.info['RepoTags'].first
  hsh[image.id] = tag unless tag == NONE_TAG
end

loop do
;; # If the current ID has a tag, render FROM instruction and break
;; # (unless this is the first command)
  if !options[:full] && commands && tags.key?(image_id)
    commands << "FROM #{tags[image_id]}"
    break
  end

  begin
    image = Docker::Image.get(image_id)
  rescue Docker::Error::NotFoundError
    abort('Error: specified image tag or ID could not be found')
  end

  cmd = image.info['ContainerConfig']['Cmd']

  if cmd && cmd.size == 3
    cmd = cmd.last

    if cmd.start_with?(NOP_PREFIX)
      commands << cmd.gsub(NOP_PREFIX,'').gsub(/^USER[ |\t]+\[(.*)\]/,'USER \1')
    else
      commands << "RUN #{cmd}".split.join(' ')
    end
  end

  image_id = image.info['Parent']
  break if image_id == ''
end

puts commands.reverse

17.17.21. image:jenkins/jenkins

  • Parent images: openjdk:8-jdk
  • GitHub
# latest LTS
docker pull jenkins/jenkins:lts
# Volume jenkins_home is created and holds Jenkins data directory includes plugins and configuration for upgrade

docker run --name myjenkins -d -v jenkins_home:/var/jenkins_home -p 8080:8080 -p 50000:50000 jenkins/jenkins:lts

# Pass JVM parameters
docker run --env JAVA_OPTS=-Dhudson.footerURL=http://mycompany.com # other options...

# config logging
mkdir data
cat > data/log.properties <<EOF
handlers=java.util.logging.ConsoleHandler
jenkins.level=FINEST
java.util.logging.ConsoleHandler.level=FINEST
EOF
docker run --env JAVA_OPTS="-Djava.util.logging.config.file=/var/jenkins_home/log.properties" # other options...

# JENKINS_OPTS
# run Jenkins behind Nginx
# mysite.com/jenkins
docker run --env JENKINS_OPTS="--prefix=/jenkins"

# https://wiki.jenkins.io/display/JENKINS/Jenkins+behind+an+NGinX+reverse+proxy

17.17.22. image:ubuntu

# as the time of writing, these lines are equivalent
docker pull ubuntu:latest
docker pull ubuntu:18.04
docker pull ubuntu:bionic
docker pull ubuntu:bionic-20181112
# The default variant for arm64 is currently v8
docker pull arm64v8/ubuntu:bionic

# run and bash in. Refer to docker:run:privileged
docker run -it --rm --privileged ubuntu:xenial /bin/bash

services:
  nodejs:
    image: ubuntu:latest

tty: true

17.17.23. image:node

17.17.23.1. Run image directly
docker run -it --rm --name my-running-script \
   -v "$PWD":/usr/src/app \
   -w /usr/src/app \
   node:8 \
   node your-daemon-or-script.js

# `node your-daemon-or-script.js` is a command run. Can be other things

npm install && npm run build

.PHONY: enter

enter:
    docker run -it --rm --name my-running-script \
       -v `pwd`:/usr/src/app \
       -w /usr/src/app \
       -p 3001:3001 \
       node:18.18 \
       bash -c "npm i -g serve && bash"
17.17.23.2. Version
  1. node:boron

    Latest version 6.x (LTS) Parent image: buildpack-deps:jessie Available: git, openssh-client

    Use docker logs -f container_name to tail log

  2. node:<version>
    18.18
    may have variants e.g. 18.18.0
    (no term)

    10.15.3 

    FROM --platform=linux/x86_64 node:10.15.3
RUN mkdir /app
WORKDIR /app
CMD ["node", "--version"]

    node-build-image:
    docker build --force-rm --no-cache -t mynode:1.0 -f Dockerfile-node .

npminstall:
    cd ../../relative/path; docker run -it --rm --platform linux/x86_64 --name mynodecontainer \
       -v `pwd`:/app \
       mynode:1.0 \
       /bin/sh -c 'cd sass; npm install'

npmrunbuild:
    cd ../../relative/path; docker run -it --rm --platform linux/x86_64 --name mynodecontainer \
       -v `pwd`:/app \
       manode:1.0 \
       /bin/sh -c 'cd sass; npm run build'
17.17.23.3. Dockerfile
FROM node:16

# Place global NPM dependencies in a non-root user directory
ENV NPM_CONFIG_PREFIX=/home/node/.npm-global
# optionally if you want to run npm global bin without specifying path
ENV PATH=$PATH:/home/node/.npm-global/bin

# Install global NPM modules
RUN npm install --global zapier-platform-cli

17.17.24. image:golang

golang:1.8 Parent image: buildpack-deps:jessie-scm Available: make Environment vars: GOPATH=/go GOROOT: usr/local/bin WORKDIR $GOPATH

docker run -it --rm --name testgo golang:1.8 /bin/bash -I to go in

Running exec hello inside container will exit the container

To download, compile and run and exit the container: 

docker run --rm golang:1.8 sh -c "go get github.com/golang/example/hello/... && exec hello"

To download the compiled hello command to host directory tmp/bin: ~docker run –rm -v /tmp/bin:/go/bin golang:1.8 go get github.com/golang/example/hello…~ Then run /tmp/bin/hello where hello is the parent directory name (For private repo, add -it for inputing username and password)

17.17.25. image:vimagick/scrapyd

https://hub.docker.com/r/vimagick/scrapyd/ https://github.com/vimagick/dockerfiles/tree/master/scrapyd 

docker pull vimagick/scrapyd
nano docker-compose.yml

version: '2'
services:
  scrapy:
    image: vimagick/scrapyd
    command: bash
    volumes:
    - "./:/code"
    working_dir: /code
    restart: always

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
    'http://quotes.toscrape.com/tag/humor/',
    ]

    def parse(self, response):
    for quote in response.css('div.quote'):
        yield {
        'text': quote.css('span.text::text').extract_first(),
        'author': quote.xpath('span/small/text()').extract_first(),
        }

    next_page = response.css('li.next a::attr("href")').extract_first()
    if next_page is not None:
        yield response.follow(next_page, self.parse)

$ docker-compose run --rm scrapy
>>> scrapy runspider stackoverflow_spider.py -o top-stackoverflow-questions.json
>>> cat top-stackoverflow-questions.json
>>> exit

17.17.26. image:yandex/clickhouse-server 

version: '3.5'

services:
  madocker:
#    ..
    networks:
  - default
  clickhouse:
    container_name: myclickhouse
    image: yandex/clickhouse-server
    ports:
  - "8123:8123"
    # user is optional, 101:101 is clickhouse:clickhouse, default is root
    user: "101:101"
    volumes:
  # refer to init.sh
  - ./setupfiles/clickhouse/:/docker-entrypoint-initdb.d
  # mapping /var/lib/clickhouse does not work on Mac 2022-03-24
  # - ./clickhouse:/var/lib/clickhouse
  # - ./clickhouse-log:/var/log/clickhouse-server # optional
    networks:
  - default

docker exec -it myclickhouse bash
# inside the container, logon
clickhouse-client

# to logon using a password and default host
clickhouse-client --password abc

# to logon to any server
clickhouse-client --host <HOSTNAME> \
                  --secure \
                  -- port 9440 \
                  --user <USERNAME> \
                  --password <PASSWORD>
17.17.26.1. Init script files
#!/bin/bash
set -e
clickhouse-client -n <<-'EOSQL'
CREATE DATABASE IF NOT EXISTS malog;
CREATE TABLE IF NOT EXISTS malog.accesslog(...);
EOSQL

17.17.27. image:ngrok/ngrok 

# docker-compose is run and it creates a default network and a service called madocker
# docker-compose.yml is in folder devdocker, so the network devdocker_default is created
# --host-header overwrites the request's Host header
# -p 3000:4040 :: http://localhost:3000 is now a web UI for debugging

docker run -it --rm -p 3000:4040 --net devdocker_default -e NGROK_AUTHTOKEN=MY_AUTH_TOKEN ngrok/ngrok http --host-header="localhost:80" madocker:80

18. Lando

All subdomains of lndo.site will be DNS publicly resolved to localhost/127.0.0.1 which requires no local /etc/hosts config.

18.1. Recipe: WordPress

cd /path/to/project
lando init
lando start
lando wp core download
  • Lando creates database container with
    • user: wordpress
    • password: wordpress
    • database: wordpress

  • Sample yaml 

    name: scef
recipe: wordpress
config:
  webroot: .
  php: '7.0' # default '7.2'
  via: nginx # default apache

# add redis
services:
  cache:
type: redis:2.8 # may just be redis
persist: true

  • For Redis, add the following in wp-config.php according to lando info for plugin wp-redis lando:service:cache 

    $redis_server = array(
'host'     => 'cache',
'port'     => 6379,
//'auth'     => '12345',
'database' => 0, // Optionally use a specific numeric Redis database. Default is 0.
);

  • Commands 

    # drop into a MySQL Shell
lando mysql
lando db-import <file>

# run php commands
lando php

18.2. Recipe: Pantheon

  • https://docs.devwithlando.io/tutorials/pantheon.html
  • https://github.com/lando/lando/tree/master/plugins/lando-pantheon/recipes/pantheon
  • Stop Docker for Windows and install Lando without Docker for Windows and Git for Windows
  • git clone from Pantheon, go to the directory and run lando init and choose pantheon and a site to create .lando.yml in project root dir
  • lando terminus auth:login --machine-token=my-machine-token
    If terminus login fail
    lando ssh --user=root and run terminus auth:login --machine-token=xxx
  • lando start
  • lando pull --database=live --files=live --code=none
  • lando pull --database=none --code=none --files=live --rsync
  • wp cache flush or lando wp cache flush
    May need to clear cache from wp-admin after lando rebuild or restart
    wp admin > Settings > clear Pantheon Page Cache
  • Fully remove all lando containers and networks
    • lando stop && lando destroy && lando poweroff
    • docker rm -f $(docker ps -aq)
    • docker network prune
    • Usually lando stop && lando destroy && lando start is enough. Can also use lando rebuild
  • lando terminus wp mysite.live user list
  • lando push -m "a message" --database=none --files=none
  • Lando creates database container with:
    user
    pantheon
    password
    pantheon
    database
    pantheon
  • the appserver container has the following .sh files in folder /helpers. Their counter parts are in docker host ~/.lando/services/config/pantheon/*.sh. Both of them are updated when lando pull is run
    • pull.sh / push.sh
    • pantheon.sh
    • Add --allow-root for wp commands in those files and run /helpers/pull.sh --database=dev --files=none --code=none directly inside the container
    • Or add a bash script wrapper wp-cli:allow-root
  • apt-get update && apt-get install nano
  • $_ENV['PANTHEON_ENVIRONMENT'] is lando

18.2.1. Push a new website to Live lando:pantheon:live

  • On local Lando
    • wp search-replace '//myweb.com' '//dev-myweb.pantheonsite.io' --dry-run
    • wp search-replace '//www.myweb.com' '//dev-myweb.pantheonsite.io' --dry-run
    • wp search-replace '//myweb.lndo.site' '//dev-myweb.pantheonsite.io --dry-run'
  • Minimize files in wp-content/uploads
  • Lando push code, db and files
  • FileZilla to push missing file to Pantheon Dev
  • On Pantheon, click Initialize Test Environment
  • click Initialize Live Environment
  • Connect to Live db and check domains. Only //live-myweb.pantheonsite.io should exist
  • Change domain DNS setting and Setup redirect pantheon:domain
  • Setup SendGrid
  • Enable scheduled backup for all environments

18.2.2. Varnish lando:pantheon:varnish

  • See version varnishd -V
  • Go to CLI varnishd, then stop start quit
  • Search log varnishlog -d -q "BerespStatus == 503"
  • Config files /etc/varnish/conf.d/*.vcl

18.3. Recipe: Drupal 7

lando init
# drupal 7

# get config information
lando info

# import db.
# Can handle uncompressed, gzipped or zipped files
lando db-import db.sql.gz

lando composer
lando db-export 'db-export.sql.gz'
lando db-import 'db-import.sql.gz'
lando drush

# MySQL Shell
lando mysql

# For postgres
lando psql

# installed PHP extensions
lando php -,

18.3.1. Landofile .lando.yml

name: my-app-name
recipe: drupal7
config:
  webroot: .
  php: '5.6' # default php:7.3
  # via: apache:2.4 # default. check with Lando apache service for the default version
  database: mariadb:10.3 # default: mysql:5.7. check with Lando mysql service for the default version
  drush: "*" # default latest. Or drush:^7 for PHP 5.3
  # drush: "*" # latest
  # drush: ^7 # latest version of Drush 7
  # drush: 8.1.15 # a specific version
  # xdebug: false # default.
  config: # custom config files. 
    database: config/my-custom.cnf
    php: config/php.ini
    server: config/server.conf # for nginx
    vhosts: config/default.conf # for apache

18.3.2. settings.php

// `lando drush` may return incorrect url. `lando info` to get the URL with or without port
$base_url = "http://mysite.lndo.site:PORT_IF_NEEDED";

$databases = array (
  'default' => 
  array (
    'default' => 
    array (
  'database' => 'drupal7',
  'username' => 'drupal7',
  'password' => 'drupal7',
  'host' => 'database',
  'port' => '3306',
  'driver' => 'mysql',
  // for postgress: username:postgres, password:'', port:5432
    ),
  ),
);

18.4. Service: phpMyAdmin

name: mysite
recipe: pantheon
config:
  framework: wordpress
  site: mysite
  id: XXX-XXX

proxy:
 pma:
   - pma.mysite.lndo.site

services:
 pma:
   type: phpmyadmin
   hosts:
     - database

18.5. Service: MariaDB

  • Pantheon recipe uses MariaDB
  • Supported versions: 10.1 (default), 10.2, 10.3
docker exec -it projname_database_1 bash
mysql

18.6. Command line

List names of apps
lando list
Destroy docker containers inside the current app
lando destroy. After that, it requires pull database, Pantheon Terminus login again.
Spin down all lando related containers
lando poweroff

18.6.1. ssh

lando ssh [appname] [service]

blank appname is current app blank service is appserver

Bash into appserver :: lando ssh --user=root

18.6.2. db-import

lando db-export cs-devops/a.sql.gz
lando db-import a.sql.gz
lando db-import a.sql

18.7. SSL

  • Enable https://yourapp.lndo.site

As of Lando 3.0.0 RC: 

# Add the Lando CA
sudo cp -r ~/.lando/certs/lndo.site.pem /usr/local/share/ca-certificates/lndo.site.pem
sudo update-ca-certificates

# Remove Lando CA
sudo rm -f /usr/local/share/ca-certificates/lndo.site.pem
sudo update-ca-certificates --fresh
Add lndo.site.pem to Chrome
Settings > Manage Certificates > Authorities > Import, select and check all options

18.8. XDebug lando:xdebug

Just add xdebug: true to a recipe's config or add it under a php service 

name: myapp
recipe: pantheon
config:
  framework: wordpress
  site: myapp
  id: abc-xyz-1234
  xdebug: true
  conf:
    # optional: load a custom config file with XDebug settings
    php: .cs-devops/php.ini
  • lando rebuild if .lando.yml is modified

If xdebug is not triggering, try appending url parameter XDEBUG_START_SESSION=LANDO

If it's still not triggered, try:

  • lando restart
  • Still not working, remove xdebug: true lando restart and set xdebug using custom php.ini
xdebug.max_nesting_level = 256
xdebug.show_exception_trace = 0
xdebug.collect_params = 0
xdebug.remote_enable = 1
xdebug.remote_autostart = 1
xdebug.remote_host = YOUR HOST IP ADDRESS

Run lando info --deep | grep IPAddress to help discover the host ip address

18.9. TS: Same app name but different locations

In one folder, lando destroy then completely remove that folder.

In the new folder, lando destroy && lando poweroff and remove all docker containers and network prune. And run lando start again.

18.10. Uninstall

lando stop
lando destroy
lando poweroff
docker rm -f $(docker ps -aq)
# or only Lando containers
docker rm -f $(docker ps --filter label=io.lando.container=TRUE --all -q)
docker network prune -f
  • Windows
    • Add/Remove Program to uninstall Lando
  • Ubuntu
    • sudo apt-get remove lando or use Software Center to uninstall
  • Remove config
    ~/.lando
    find config location lando config | grep userConfRoot
    (no term)
    Windows C:\Users\myname\.lando

19. Jenkins

19.1. Install

Ubuntu or Debian-based 

wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install jenkins

It will:

  • Setup Jenkins as a daemon launched on start. See /etc/init.d/jenkins for more details.
  • Create a jenkins user to run this service.
  • Direct console log output to the file /var/log/jenkins/jenkins.log. Check this file if you are troubleshooting Jenkins.
  • Populate /etc/default/jenkins with configuration parameters for the launch, e.g JENKINS_HOME
  • Set Jenkins to listen on port 8080. Access this port with your browser to start configuration.
  • If your /etc/init.d/jenkins file fails to start Jenkins, edit the /etc/default/jenkins to replace the line -—HTTP_PORT=8080-— with -—HTTP_PORT=8081-— Here, "8081" was chosen but you can put another port available.
  • sudo /etc/init.d/jenkins restart

Usage: /etc/init.d/jenkins {start|stop|status|restart|force-reload}`

19.2. config

nano /etc/default/jenkins Disable DNS Multicast feature which will blow up jenkins log 

Change this: JAVA_ARGS="-Djava.awt.headless=true"
To this: JAVA_ARGS="-Djava.awt.headless=true -Dhudson.DNSMultiCast.disabled=true"
Restart
service jenkins restart

Or you can keep that feature on but just don't log

Manage Jenkins -> System Log -> Log Levels (on the left) Add the following entry:

Name: javax.jmdns Level: off

19.3. Builds

Always use sudo to run commands

20. Go

20.1. Basics, Command

Go is just an executable script file. Uninstall on Windows, go to Control Panel if you install using msi On Linux, uninstall by just replacing the bin folder

Install on Windows using zip, extract it to C:\go so C:\go\bin exists. Add C:\go\bin to PATH. GOPATH is C:\Users\username\go See phpstorm:golang 

echo $GOPATH
go version

# godoc <package_name>
godoc fmt

gofmt main.go
# -w write file
gofmt -w main.go

# Format all files in directory
go fmt ./...

# List environment variables
go env

# GOPATH :: e.g. "/root/go" on linux, where downloaded modules/packages are saved.
# GOROOT = usr/local/go
# GOARCH :: e.g. "arm64"
# GOOS :: like "linux"
# cat $GOROOT/src/go/build/syslist.go

# Compile packages and dependencies based on 1 go file
go build main.go

# A package file name should be only lowercase letters

# Folder structure in GOPATH directory
# ./bin holds executables (compiled
# ./pkg
# ./src holds packages. In this folder, run go install
# to compile and executables in ./bin
# Change GOOS and GOARCH then go install again

# Standard library including builtin package are under $GOROOT/src, e.g. $GOROOT/src/fmt
# Or go online https://golang.org/pkg/fmt/

In Go Playground, the system time is always the same http://play.golang.org

Supported IDEs

20.1.1. go run -race *.go

-race is to detect data race ./compare.go 

package main
import (...)
func f1
func f2

./main.go 

package main
import (...)
func main() {
  f1()
}

20.2. Syntax, Program Flow

Opening and closing brackets have to be in one line 

// for ... {
// } else if x ==0 {
// } else {

Define variable in block 

if x :=-42; x < 0 {
  // x will only be available in the if block
}

Switch 

import (
  "math/rand"
  "time"
)

rand.Seed(time.Now().Unix())

result := ""

switch dow := rand.Intn(6) + 1; dow {
// between 0 and 6. Result is between 1 and 7
  case 1:
    result = "Sunday!"
  case 7:
    result = "Saturday!"
  default:
    result = "Weekday!"
}
fmt.Println(result)

// Conditional expression in switch!
x := -42;
switch {
  case x < 0:
    result = "Less than zero"
  case x == 0:
    result = "Equal to zero"
  default:
    result = "Greater than zero"
}
// By default, it breaks and goes to the end of switch if there's one case matched.
// break is not necessary but `fallthrough` replaces `break` in an opposite way

for 

sum = 0
for i := 0; i < 10; i++ {
  sum += i
}

for sum < 1000 {
 sum += sum
 fmt.Println(sum)
 if sum > 200 {
   goto endofprogram
 }
}

endofprogram : fmt.Println("end of program")
// Use `break` or `continue` statements just like other language

20.3. Standard Library of packages

20.3.1. builtin

http://golang.org/pkg/builtin/

20.3.1.1. new(), make()

new() :: Allocate but not initial memory. Results in 0 storage but return a memory address make() :: Allocate and initialize memory. Results in non-zeroed storage and return a memory address

Zeroed storage means it can't accept value

Initialize complex objects before adding values. Declarations without make() can cause a panic 

m := make(map[string]int)
m["key"] = 42
// map[key:42]

Memory is deallocated by garbage collector (GC).

GC clears objects that are out of scope or set to nil

20.3.2. fmt

20.3.2.1. Output
package main
import (
  fmt
)
func main() {
  str1 := "one"
  str2 := "two"
  aNumber := 42
  // var aNumber int
  // default aNumber is zero
  // Implicit typing using :=
  isTrue := true

  // Explicit: const anInteger int = 42
  // Implicit const aString = ".."

  fmt.Println(str1, str2) // "one two"

  stringLength, err := fmt.Println(str1, str2)
  // function can return multiple values, a string and an error

  if err == nil {
    fmt.Println("String Length", stringLength)
  }

  // Without the if err == nil, it will throw an error because error is not defined
  // Use _ to ignore the err variable even if it's not returned
  stringLength, _ := fmt.Println(str1, str2)
  fmt.Println("String Length", stringLength)

  fmt.Printf("Value of aNumber: %v\n", aNumber)
  fmt.Printf("Value of isTrue: %v\n", isTrue)
  fmt.Printf("Value of aNumber as float: %.2f\n", float64(aNumber)) // 42.00
  fmt.Printf("Data types: %T, %T, %T, %T",
     str1, str2, aNumber, isTrue
  )
  // string, string, int, bool


  // Return a string
  myString := fmt.Sprintf("data types: %T, %T, %T, %T", str1, str2, aNumber, isTrue)
  fmt.Print(myString)
}

%v :: only values %+v :: add field names when printing struct %#v :: add field names and name of the struct type when printing struct %T :: type %% :: literal %

Sprintf(format string, a …interface{}) string :: format and return a string Use this to convert a struct into string str := fmt.Sprintf("%#v", val)

Fprintf(w io.Writer, format string, a …interface{}) (n int, err error)

20.3.2.2. Input
import ("fmt")

func main() {
  var s1,s2 string
  fmt.Scanln(&s1, &s2)
  // Scan one line from terminal input, demilited by space
  fmt.Println(s1, s2)
}

import (
  "fmt"
  "bufio"
  "os"
  "strconv"
  "strings"
)

func main() {
  // := is to declare and assign value to a new variable
  reader := bufio.NewReader(os.stdin)
  fmt.Print("Enter text: ")
  str, _ := reader.ReadString('\n')
  // Single quote means it's a byte not a string
  fmt.Print(str)
  // one two three

  fmt.Print("Enter text: ")
  str, _ = reader.ReadString('\n')
  f, err := strconv.ParseFloat(strings.TrimSpace(str), 64)
  if err != nill {
    fmt.Println(err)
  } else {
    fmt.Println("Value of number: ", f);
  }
}

20.3.3. strings

strings.ToUpper(str) strings.Title(str) / Initial capped strings.EqualFold(str1, str2) / case-insensitive compare / default == is case-sensitive strings.Contains(str, "exp") / true/false

20.3.4. math

import (
  "fmt"
  "math/big"
  "math"
)
i1, i2, i3 := 12, 45, 68
intSum := i1 + i2 + i3 // Good!

f1, f2, f3 := 23.5, 65.1, 76.3
floatSum := f1 + f2 + f3 // Not good

var b1, b2, b3, bigSum big.Float
b1.SetFloat64(23.5)
b2.SetFloat64(65.1)
b3.SetFloat64(76.3)
bigSum.Add(&b1, &b2).Add(&bigSum, &b3)
fmt.Printf("BigSum = %.10g\n", &bigSum)

circleradius := 15.5
circumference := circleRadius * math.Pi
fmt.Printf("Circumference: %.2f\n", circumference)

20.3.5. time

import (
  "fmt"
  "time"
)

t := time.Date(2009, time.November, 10, 23, 0,0,0, time.UTC)
now := time.Now()
fmt.Printf("Go launched at %s\n", t)
now.Month()
now.Day()
now.Weekday()

tomorrow := now.AddDate(0,0,1)

longFormat := "Monday, January 2, 2006"
fmt.Println("Tomorrow is ", tomorrow.Format(longFormat))

shortFormat := "1/2/06"

t := time.Now()

fmt.Printf("%d %02d %02d %02d %02d %02d\n",
t.Year(), t.Month(), t.Day(),
t.Hour(), t.Minute(), t.Second())

start := time.Now()
// ...
end := time.Now()
delta := end.Sub(start)
// Or
fmt.Println(time.Since(start).Seconds())

20.3.7. io, os, path

import (
  "io"
  "os"
  "io/ioutil"
  "path/filepath"
)

func main() {
  content := "Hello from Go!"
  file, err :=os.Create("./fromString.txt")
  checkError(err)

  defer file.Close()

  ln, err := io.WriteString(file, content)

  checkError(err)

  fmt.Printf("All done with file of %v characters", ln)

  // convert a string to binary
  bytes := []byte(content)
  // 0644 is file permission
  ioutil.WriteFile("./fromBytes.txt", bytes, 0644)

  // Read a file
  fileName := "./hello.txt"
  // read as binary
  content, err := ioutil.ReadFile(fileName)
  checkError(err) 
  // binary/byte
  fmt.Println(content)
  result := string(content)
  fmt.Println(result)

  // Reader a directory
  root, _ := filepath.Abs(".")

  fmt.Println(root)

  err := filepath.Walk(root, processPath)
  checkError(err)

}

func checkError(err error) {
  if err != nil {
    // Stop if there's an error
    panic(err)
  }
}

func processPath(path string, info os.Fileinfo, err error) error {
  if err != nil {
    return err
  }

  // not root
  if path != "." {
    if info.IsDir() {
      fmt.Println("Directory:", path)
    } else {
      fmt.Println("File:", path)
    }
  }

  return nil
}

Print OS environments 

// To set a key/value pair, use `os.Setenv`. To get a
// value for a key, use `os.Getenv`. This will return
// an empty string if the key isn't present in the
// environment.
os.Setenv("FOO", "1")
fmt.Println("FOO:", os.Getenv("FOO"))
fmt.Println("BAR:", os.Getenv("BAR"))

// Use `os.Environ` to list all key/value pairs in the
// environment. This returns a slice of strings in the
// form `KEY=value`. You can `strings.Split` them to
// get the key and value. Here we print all the keys.
fmt.Println()
for _, e := range os.Environ() {
     pair := strings.Split(e, "=")
     fmt.Println(pair[0])
}

20.3.8. path/filepath

filepath.Walk("../images/", func(path string, info os.FileInfo, err error) {
  if info.IsDir() {
    return nil
  }
  fmt.Println(path)
  return nil
})

20.3.9. encoding/json, encoding/csv

Refer to golang:net/http:json 

json.NewDecoder(r io.Reader) *Decoder
json.Unmarshal(data []byte, v interface{}) error

// convert any json string to struct
import ("encoding/json" "fmt" "reflect") 

var input = `
{
  "response_type": "in_channel",
  "text": "hello text",
  "attachments": [
    { "text": "attachment text" }
  ]
}
`
var val map[string]interface{}

if err := json.Unmarshal([]byte(input), $val); err != nil {
  panic(err)
}

fmt.Println(val)
// map[response_type:in_channel text:hello text attachments:[map[text:attachment text]]]
for k, v := range val {
  fmt.Println(k, reflect.TypeOf(v))
}

import (
 "os"
 "encoding/csv"
)

f, err := os.Open("../abc.txt")
checkError(err)
defer f.Close()

rdr := csv.NewReader(f)
rdr.Comma = '\t'
rdr.TrimLeadingSpace = true
rows, err := rdr.ReadAll()
checkError(err)
for i, row := range rows {
  fmt.Println(row[i])
}

20.3.10. log

Output to stdout 

import ( "log" )
log.Println(string(body))

Write to os.Stdout os.Stderr 

import (
"io"
"io/ioutil"
"log"
"os"
)
var (
  Trace *log.Logger
  Info *log.Logger
  Warning *log.Logger
  Error *log.Logger
)
func Init(
  traceHandle io.Writer,
  infoHandle io.Writer,
  warningHandle io.Writer,
  errorHandle io.Writer) {
  Trace = log.New(traceHandle, 
     "TRACE: ",
     log.Ldate|log.Ltime|log.Lshortfile)

  Info = log.New(infoHandle, 
     "INFO: ",
     log.Ldate|log.Ltime|log.Lshortfile)

  Warning = log.New(warningHandle, 
     "WARNING: ",
     log.Ldate|log.Ltime|log.Lshortfile)

  Error = log.New(errorHandle, 
     "ERROR: ",
     log.Ldate|log.Ltime|log.Lshortfile)

  // log.New(out io.Writer, prefix string, flag int)
}

func main() {
  // system devices that support io.Writer interface
  Init(ioutil.Discard, os.Stdout, os.Stdout, os.Stderr)

  // write to stdout
  Info.Pringln("...")
}

Write to file 

file, err := os.OpenFile("file.txt", os.O_CREATE|os.WRONLY|os.O_APPEND, 0666)
if err != nil {
  log.Fatalln("Failed to open log file", output, ":", err)
}

multi := io.MultiWriter(file, os.Stdout)

MyFile := log.New(multi,
       "PREFIX: ",
       log.Ldate|log.Ltime|log.shortfile)

log.Fatalln("…") same as Println() but followed by a call to os.Exit(1) log.Panicln() same as Println() but followed by panic()

20.3.11. os/exec

20.3.11.1. Basics
import (
  "fmt"
  "os"
  "os/exec"
)

func main() {
  cmd := "curl"
  args := []string{"https://..."}

  /* if command contains wildecard *
     exec.Command("/bin/sh", "-c", "mv ./source_dir/* ./dest_dir")
   */

  // Simply run
  if err := exec.Command(cmd, args...).Run(); err != nil {
    fmt.Println(os.Stderr, err)
    os.Exit(1)
  }
  fmt.Println("Success!")
  // Simply run.

  // Run and catch STDOUT
  var (
    cmdOut []byte
    err error
  )

  if cmdOut, err = exec.Command(cmd, args...).Output(); err != nil {
    fmt.Fprintln(os.Stderr, "An error: ", err)
    os.Exit(1)
  }

  result := string(cmdOut)
  // Run and catch STDOUT

  // Catch output line by line as a stream
  // Use cmd.Start() and cmd.Wait()
  // Catch stdout by adding pipe

  cmd2 := exec.Command(cmd, args...)
  cmdReader, err := cmd2.StdoutPipe()
  if err != nil {
    fmt.Fprintln(os.Stderr, "Error creating StdoutPipe for cmd", err)
    os.Exit(1)
  }

  // import "bufio"
  scanner := bufio.NewScanner(cmdReader)
  go func() {
    for scanner.Scan() {
      fmt.Printf("Output header | %s\n", scanner.Text())
    }
  }()

  err = cmd2.Start()
  if err != nil {
    fmt.Fprintln(os.Stderr, "Error starting Cmd", err)
  }

  err = cmd2.Wait()
  if err != nil {
    fmt.Fprintln(os.Stderr, "Error waiting for Cmd", err)
    os.Exit(1) 
  }
  // Catch output line by line as a stream.
}
20.3.11.2. SSH command
type SSHCommander struct {
        User string
        IP   string
}

func (s *SSHCommander) Command(cmd ...string) *exec.Cmd {
        arg := append(
                []string{
                        fmt.Sprintf("%s@%s", s.User, s.IP),
                },
                cmd...,
        )
        return exec.Command("ssh", arg...)
}

func main() {
        commander := SSHCommander{"root", "50.112.213.24"}

        cmd := []string{
                "apt-get",
                "install",
                "-y",
                "jq",
                "golang-go",
                "nginx",
        }

        if err := commander.Command(cmd...); err != nil {
                fmt.Fprintln(os.Stderr, "There was an error running SSH command: ", err)
                os.Exit(1)
        }
}
20.3.11.3. Cmd

Customize a command before it gets run by Run, Output or CombinedOutput methods 

cmd := "git"
args := []string{"status"}

cmd2 := exec.Command(cmd, arg...)
// Working directory otherwise is the process's current dir
cmd2.Dir = "/home/user/"

20.3.12. net/http

20.3.12.1. HTTP request
resp, err := http.Get("http://example.com/")
...
resp, err := http.Post("http://example.com/upload", "image/jpeg", &buf)
...
resp, err := http.PostForm("http://example.com/form",
        url.Values{"key": {"Value"}, "id": {"123"}})

import (
  "io/ioutil"
  "net/http"
  "encoding/json"
  "strings"
  "math/big"
)

type Tour struct {
  Name, Price string
}

func main() {
  url := "http://..."
  resp, err := http.Get(url)
  checkError(err)
  fmt.Printf("Response type: %T\n", resp)

  // Important!!!
  defer resp.Body.Close()

  bytes, err := ioutil.ReadAll(resp.body)
  checkError(err)
  content := string(bytes)
  fmt.Pritnln(content)

  tours := toursFromJson(content)
  fmt.Println(tours)

  for _, tour := range tours {
    // price is a string and convert to float
    price, _, _ := big.ParseFloat(tour.Price, 10, 2, big.ToZero)
    fmt.Printf("%v $%.2f\n", tour.Name, price)
  }
}

func toursFromJson(content string) []Tour {
  tours := make([]Tour, 0, 20)

  decoder := json.NewDecoder(strings.NewReader(content))

  _, err := decoder.Token()
  checkError(err)

  var tour Tour
  for decoder.More() {
    err := decoder.Decode(&tour)
    checkError(err)
    tours = append(tours, tour)
  }

  return tours
}

HTTP POST json 

tmp := `{"text": "delayed response", "attachments": [{ "text": "test"}]}`
// or
// tmp, err := json.Marshal(myStruct)
req := bytes.NewBuffer([]byte(tmp))
resp, err := http.Post(sc.ResponseURL, "application/json", req)
checkError(err)
body, _ := ioutil.ReadAll(resp.Body)
fmt.Println(string(body))
20.3.12.2. HTTP server
  1. Use http.Handle
    import (
  "fmt"
  "net/http"
  "html/template"
)

// default Handler
type Hello struct{}

// Implement handler that is an interface
func (h Hello) ServeHTTP(w http.ResponseWriter, r *http.Request) {
  fmt.Fprintf(w, "<h1>hello</h1>")
}

// func Handle(pattern string, handler Handler)
// type Handler interface {
//  ServeHTTP(resp, *request)
// }

func main() {
  var h Hello
  err := http.ListenAndServe(":4000", h)
  // addr string, handler Handler
  // if handler is nil, DefaultServeMux is used
  // or localhost:4000 (only localhost:4000 listens. multiple domain, non localhost, can be resolved to the same server)
  checkError(err)
}
  2. Use http.HandleFunc
    // HandleFunc add handlers to DefaultServeMux
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
  fmt.Fprintf(w, "hello html")
})

fmt.Println(http.ListenAndServe(":4000", h))
checkError(err)
  3. Template
    import "html/template"

type Page struct {
  Name string
}

func main() {
  templates := template.Must(template.ParseFiles("templates/index.html"))

  http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    p := Page{Name: "Gopher"}

    if err := templates.ExecuteTemplate(w, "index.html", p); err !=  nil {
      http.Error(w, err.Error(), http.StatusInternalServerError)
    }
  })
}

    index.html 

    {{.Name}}

    If url.name is defined, then that will be displayed

  4. Read POST data
    1. Content-Type json golang:net/http:json
      type test_struct struct {
  Test string
  IsNew bool // JSON booleans
  Age float64 // JSON numbers
  myArray []interface{} // JSON arrays
  myObj map[string]interface{} // JSON objects
  isNull nil // JSON null
}

func test(w http.ResponseWriter, r *http.Request) {
  // Read body as a whole
  body, err := ioutil.ReadAll(r.Body)
  checkError(err)
  log.Println(string(body))

  //decoder := json.NewDecoder(r.Body)
  var t test_struct
  err = json.Unmarshal(body, &t)
  checkError(err)
  fmt.Fprintf(w, t.Test)

  // Get Header
  if a := r.Header.Get("x-hub-signature"); len(a) == 0 {
    panic()
  }

}

func main() {
  http.HandleFunc("/test", test)
}

      See 15.20.1.5

    2. Content-Type:application/x-www-form-urlencoded
      r.ParseForm()
r.Form.Get("key")
res := r.FormValue("<your param name>")
    3. Return Json as application/json
      func testjson(w http.ResponseWriter, r *http.Request) {
  body, err := ioutil.ReadAll(r.Body)
  checkError(err)
  type slackResponse struct {
    ResponseType string `json:"response_type"`
    Text string `json:"text"`
    Attachments []map[string]string `json:"attachments"`
  }
  a2 := []map[string]string{map[string]string{"text": string(body)}}
  resp := slackResponse{"in_channel", a2}
  js, err := json.Marshal(resp)
  checkError(err)
  w.Header().Set("Content-Type", "application/json")
  w.Write(js)

  /* Or you can just output a json string
  var p = `{
  "response_type": "in_channel",
  "attachments": [
     {  "text": "%s" }
   ]
}`
   resp := fmt.Sprintf(p, string(body))
   w.Header().Set("Content-Type", "application/json")
   fmt.Fprintf(w, resp)
  */

}
func main() {
  http.HandleFunc("/json", testjson)
}

      More complicated value `attachments` 

      // this is the json to construct
// {"response_type":"in_channel","text":"hello text","attachments":[{"text":"attachment text"},{"text2":{"name":"hello"}}]}

type Profile struct {
        Response_type string `json:"response_type"`
        Text          string `json:"text"`
        Attachments []interface{} `json:"attachments"`
}

func main() {
        a2 := make([]interface{}, 2)
        a2[0] = map[string]string{"text": "attachment text"}
        a2[1] = map[string]map[string]string{"text2": map[string]string{"name": "hello"}}
        profile := Profile{"in_channel", "hello text", a2}
        js, err := json.Marshal(profile)
        checkError(err)
        fmt.Println(string(js))
        str := fmt.Sprintf("%#v", profile)
        fmt.Println(str)
}

20.4. Custom package, Third Party Package

For third-party packages, godoc.org and github.com/avelino/awesome-go https://www.lynda.com/Go-tutorials/What-you-should-know-before-watching-course/439416/472998-4.html?srchtrk=index%3a1%0alinktypeid%3a2%0aq%3ago+language%0apage%3a1%0as%3arelevance%0asa%3atrue%0aproducttypeid%3a2 $GOPATH/src/mypkg 

package main
// define a package name.

// main() is the starting point in go run
func main() {
  n1, l1 := FullName("Zaphod", "Beeblerox")
}

// captital P means this method is public instead of private in this package
func Println() {
}

func addValues(value1 int, value2 int) int {
// can also be value1, value2 int
  return value1 + value2
}

func addAllValues(values ...int) int {
  sum := 0
  for i := range values {
    sum += values[i]
  }
  return sum
}

func FullName(f, l string) (string, int) {
  full := f + " " + l
  length := len(full)
  return full, length
}

func FullNameNakedReturn(f, l string) (full string, length int) {
  // The returned variables have already been declared
  // Don't use :=
  full = f + " " + l
  length = len(full)
  return
}

;; # mypkg.Println(...)

import (
  // github.com/username/reponame/foldername/example.go
  "github.com/username/reponame/foldername"
  // alias
  stdimage "image"
)

All packages and codes should be under one go workspace which the folder at GOPATH. Assuming GOPATH is ~/go in Windows installation. 

cd ~/go
mkdir -p src/github.com
git clone ../gitusername/myreponame
~/go/src/github.com/gitusername/reponame/pkg1/pkg1.go
~/go/src/github.com/gitusername/reponame/pkg2/pkg2.go

;; # build all necessary packages
go get github.com/gitusername/reponame/...
;; # ~/go/bin/pkg1 (executable)
;; # ~/go/bin/pkg2 (executable)

;; # Build one package
go get github.com/gitusername/reponame/pkg1/...
;; # ~/go/bin/pkg1 (executable)

;; # At ~/go/src/github.com/gitusername/reponame, run this to see if there's compile error
go build ...

;; # At ~/go/src/github.com/gitusername/reponame, run this to format all .go files
go fmt ./...

20.5. Types

20.5.1. Basic

float32 float54 complex64 complex128 Integers: int8 int16 int32 int64 only positive numbers and zero (unsigned): uint8 uint16 uint32 uint64 byte uint int uintptr

Data collections: Arrays Slices Maps Structs

Language organization: Functions Interfaces Channels

Data management Pointers

Type conversion 

var int1 int = 5
var float1 float64 = 42
var s := `{ "votes": { "option": "3"} }`
sum := float64(int1) + float1
// convert string to byte
bytevar := []byte(s)

// convert string array to string
stringArray := []string{"Hello","world","!"}
justString := strings.Join(stringArray," ")

// Convert string to float golang:strconv
f, err := strconv.ParseFloat(strings.TrimSpace(str), 64)

// Delcare only for multiple variables
var (
  count int
  sum float64
)

var a, b string
a, b := "", ""

20.5.2. array

Array can't be sorted or resized 

var colors [3]string
colors[0] = "Red"
colors[1] = "Green"
fmt.Println(colors)
// [Red Green]
fmt.Println(colors[1])
// Green

// Define in one line
var numbers = [5]int{5,3,1,2,4}

// Array length
// len(colors)

for i := 0; i < len(colors); i++ {
  fmt.Println(colors[i])
}

for i := range colors {
  fmt.Pritnln(colors[i])
}

20.5.3. slice

slice is built on top of array 

var colors = []string{"Red", "Green", "Blue"}
fmt.Println(colors)
// The same as array

// Add item to end of slice
colors = append(colors, "Purple")
// Remove the first item in slice
colors = append(colors[1:len(colors)])
// Below is the same to remove the firs item in slice
// as the default on the right of : is the length of the slice
colors = append(colors[1:])
// Remove the last item in slice
// the default on the left of : is 0
colors = append(colors[:len(colors)-1])

// Declare a slice using make
numbers := make([]int, 5, 10)
// 5 is inital size, 10 is capacity (optional)
// Current capacity cap(numbers)
// Define 5 values, then add another item, 
numbers = append(numbers, 123)
// cap(numbers) is 2+10 = 12

// import("sort")
sort.Ints(numbers) // ascending

20.5.4. map

Map is an unorderd collection of key-value pairs 

states := make(map[string]string)
// empty: map[]
states["WA"] = "Washington"
states["OR"] = "Oregon"
// map[WA:Washington OR:Oregon]

delete(states, "OR")

for k, v := range states {
  fmt.Printf("%v: %v\n", k, v)
}

// create a slice to hold keys of a map
keys :=make([]string, len(states))
i := 0
for k := range states {
  keys[i] = k
  i++
}

sort.Strings(keys)

for i := range keys {
  fmt.Println(states[keys[i]])
}

20.5.5. struct, interface

Struct is a data structure. Similar to Java's classes: encapsulate data and methods. No inheritance.

Every value in Go is an instance of a type, and every type is an implementation of at least one interface, the interface without any methods.

You may see function like this. 

func Errorf(format string, a ...interface{}) {}

// ...interface{} means it can be multiple values of all types

type Dog struct {
  Breed string
  Weight int
  Sound string
}

// Define method for a Struct
func (d Dog) Speak() {
  // d is a reference (copy) of the original object
  // changing d here doesn't affect the variable in global scope
  fmt.Println(d.Sound)
}

func (d *Dog) SpeakThreeTimes() {
  // d is a point to the original object
  // Changing d here also changes the variable in global scope
  d.Sound = fmt.Sprintf("%v! %v! %v!", d.Sound, d.Sound, d.Sound)
  fmt.Println(d.Sound)
}

// Define an interface
type Animal interface {
  SpeakWord() string
}

func (d Dog) Speak() string {
  return "Woof!"
}

// Attach interface to other class

type Cat struct {
  // ...
}

func (c Cat) Speak() string {
  return "Meow!"
}

func main() {
  poodle := Dog{"Poodle", 34, "Woof"}
  // fmt.Println(poodle)
  // {Poodle, 34, Woof}

  // dump field name and values
  fmt.Printf("%+v\n", poodle)
  // {Breed:Poodle Weight:34 Sound:Woof}

  poodle.Sound = "Arf"

  poodle.Speak()

  poodle2 := Animal(Dog{})
  // This dog has an interface!
  // Or this dog is converted to interface Animal
  fmt.Println(poodle2)

  // Attach interface to multiple objects of different types
  animals := []Animal{Dog{}, Cat{}}
  for _, animal := range animals {
    // _ means to throw away the index
    fmt.Println(animal.Speak())
  }

}

20.5.6. pointer

// craete a pointer
var p *int
if p != nil {
  fmt.Println(*p)
} else {
  fmt.Println("p is nil")
}

var v int = 42
p = &v
// explict
var value1 float64 = 42.13
pointer1 := &value1
fmt.Println(*p)

*pointer1 = *pointer1 / 42
// both value1
fmt.Println(*pointer, value1)

20.5.7. error

import "errors"

myError := errors.New("My error string")
fmt.Println(myError)
// str := myError.Error()

attendance := map[string]bool{
  "Ann": true,
  "Mike": true}

attended, ok := attendance["Mike"]
if ok {
  // do things  
} else {
  // return error
}

20.6. Concurrency


// Avoid race condition
var mu sync.Mutex
var wg sync.WaitGroup
wg.Add(len(paths))

for _, path := range paths {
  go func(path string) {
    pixels := getPixels(path)
    mu.Lock()
    {
      images = append(images, pixels)
    }
    mu.Unlock()
    wg.Done()
  }(path)
}

20.7. Structure

;; # $GOPATH
./src/libraries
./src/palindrome
./src/stringutil

;; # Go to the directory that has a go file that has main() function
;; # and run go install

Defer defer keyword only works within a function Deferred statements are FILO and are run at the end of the current function, rather than waiting for the entire application Good for disconnect database 

defer fmt.Println("Statement 1")
defer fmt.Println("Statement 2")
defer fmt.Println("Statement 3")
defer fmt.Println("Undeferred statement")
// Undeferred statement, 3, 2, 1

20.8. Shrink compiled binary file size

;; # same as go get, go install
go build -ldflags="-s -w" cmd/go

It strips the DWARF tables needed for debugging, but keeps the annotations needed for stack traces (also panic as well)

Then use linux:upx to further compress it.

20.9. Project

20.9.1. GitHub Webhook go:github-webhook

#+NAME main.go 

package main

import (
	"fmt"
	"net/http"
	"io/ioutil"
	"os"
	"os/exec"
	"log"
	"crypto/hmac"
	"crypto/sha1"
	"encoding/hex"
	"encoding/json"
	"errors"
	"strings"
)

type test_struct struct {
	Repository map[string]interface{}
}

type GitHubHookContext struct {
	Signature string
	Event string
	Id string
	Payload []byte
}

type CIFolder struct {
	repo string
	Path string
}

func checkError(err error) {
	if err != nil {
		panic(err)
	}
}

func signBody(secret, body []byte) []byte {
	computed := hmac.New(sha1.New, secret)
	computed.Write(body)
	return []byte(computed.Sum(nil))
}

func verifySecret(secret []byte, signature string, body []byte) bool {

	const signaturePrefix = "sha1="
	const signatureLength = 45 // len(SignaturePrefix) + len(hex(sha1))

	if len(signature) != signatureLength || !strings.HasPrefix(signature, signaturePrefix) {
		return false
	}

	actual := make([]byte, 20)
	hex.Decode(actual, []byte(signature[5:]))

	return hmac.Equal(signBody(secret, body), actual)
}

func ParseGitHubHookContext(secret []byte, r *http.Request) (*GitHubHookContext, error) {

	hc := GitHubHookContext{}

	if hc.Signature = r.Header.Get("x-hub-signature"); len(hc.Signature) == 0 {
		return nil, errors.New("No signature!")
	}
	log.Println("Signature: ", hc.Signature)

	if hc.Event = r.Header.Get("x-github-event"); len(hc.Event) == 0 {
		return nil, errors.New("No event!")
	}
	log.Println("Event: ", hc.Event)

	if hc.Id = r.Header.Get("x-github-delivery"); len(hc.Id) == 0 {
		return nil, errors.New("No event Id!")
	}
	log.Println("ID: ", hc.Id)

	body, err := ioutil.ReadAll(r.Body)
	if err != nil {
		return nil, err
	}

	if !verifySecret(secret, hc.Signature, body) {
		return nil, errors.New("Invalid signature")
	}

	hc.Payload = body
	return &hc, nil
}

func pullGitHub(path string) {
	var args []string
	args = []string{"reset", "--hard"}
	runCommands(path, "git", args, "git reset error: ")
	args = []string{"clean", "-df"}
	runCommands(path, "git", args, "git clean error: ")
	args = []string{"checkout", "master"}
	runCommands(path, "git", args, "git checkout master error: ")
	args = []string{"pull"}
	runCommands(path, "git", args, "git pull error: ")
}

func runCommands(path string, cmd string, args []string, msg string) {
	var (
		cmdOut []byte
		err error
	)
	outputArgs := strings.Join(args, " ")
	log.Println("Running ", cmd, outputArgs)
	actual := exec.Command(cmd, args...)
	actual.Dir = path
	if cmdOut, err = actual.Output(); err != nil {
		log.Println(os.Stderr, msg, err)
		os.Exit(1)
	}

	log.Println(string(cmdOut))
}

func main() {
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		secret := os.Getenv("GB_WEBHOOK_KEY")
		hc, err := ParseGitHubHookContext([]byte(secret), r)
		if err != nil {
			log.Printf("Failed processing hook! ('%s')", err)
			panic(err)
		} else {

			if hc.Event == "push" {
				//log.Println(string(hc.Payload))
				var t test_struct
				err = json.Unmarshal(hc.Payload, &t)
				checkError(err)

				switch reponame := t.Repository["full_name"]; reponame {
				case "NewcomDev/nodejs":
					pullGitHub("/home/newcom/www/nodejs-dev")
				}

			}
		}

		fmt.Fprintf(w, "hello html")
		//out := fmt.Sprintf("%#v", r.URL)
		//fmt.Fprintf(w, out)
	})

	fmt.Println(http.ListenAndServe(":49162", nil))
}

sudo docker run -it --rm --name testgo -v /home/newcom/digitalocean/docker/newcomgo:/go golang:1.8 /bin/bash -I
# Inside Golang
go build main.go

# Go to project folder
./main

20.10. To Learn

Concurrency in Go Goroutine

  • a lightweight synchronized thread managed by the runtime

Channel

  • a typed conduit for msgs between goroutines

Select

  • Lets a goroutine wait for multiple communication operations

Web Frameworks (REST)

  • Beego, Martini, Gorilla, Gocraft, Revel

Database Drivers

21. Vagrant

21.1. Install

Install Oracle VM Virtualbox first
https://www.virtualbox.org/wiki/Downloads
VirtualBox Oracle VM VirtualBox Extension Pack is needed
https://www.vagrantup.com/docs/virtualbox/
(no term)
VirtualBox version 5.0.x and 5.1.x are supported by Vagrant 1.x
(no term)
Simply install the latest version of VirtualBox to upgrade
(no term)

TS: Error: "VT-x/AMD-V hardware acceleration is not available"

  • On Windows, check if Hyper-V or Windows Hypervisor Platform is on by going to Windows Features
  • Run optionalfeatures.exe to go directly to Windows Features
  • Turn it off. As Windows takes over the hardware acceleration
  • Then reboot and change UEFI or BIOS setting
    • name might be "Intel VT-x, VT-d, Virtualization Extensions, Vanderpool" under Chipset, Northbridge or CPU configuration
(no term)
Install vagrant:plugin
Hypervisor
software, firmware or hardware that creates and runs virtual machines. May be called virtualization
Hyper-V
Windows' hypervisor
  • Docker for Windows requires Hyper-V on Windows
  • Windows Subsystem for Linux v2 (WSL2) supports docker using Hyper-V even on Windows Home edition
Virtualbox hypervisor
does not work with Windows having Hyper_V enabled
  • Inside Virtualbox, cannot have another virtualization
vagrant --version
# 1.x
where vagrant

# download image and create Vagrantfile
vagrant init hashicorp/precise64

# Ubuntu 18.04
vagrant init hashicorp/bionic64

vagrant up && vagrant ssh

21.2. Vagrant VM and VirtualBox

# Status of all Vagrant VM's
vagrant global-status --prune

# List all running VirtualBox VMs
# vboxmanage.exe is not in PATH, change directory
cd /c/Program Files/Oracle/VirtualBox
./VBoxManage.exe list runningvms

# Show a VM info
./VBoxManage.exe showvminfo myvm1

# VirtualBox version
./VBoxManage.exe --version
  • Forward ports after Vagrant has started in VirtualBox
    • Select the running VM in VirtualBox, Settings > Network > Port Forwarding, add e.g. forward 127.0.0.1:2223 to 10.0.2.15:22
  • By default, VirtualBox does not enable symlink for shared folders for hosts that don't support symlink
    • Hosts such as Windows
    • On Windows, enable 3
    • Run this on Windows ./VBoxManage.exe setextradata livagrantdev VBoxInternal2/SharedFoldersEnableSymlinksCreate/vagrant 1
    • vagrant halt, vagrant up
    • Still not possible to create symlinks ln -s path/to/source path/to/symlink inside Vagrant but any symlinks created on Windows using mklink work as intended on Vagrant

21.3. Frequent commands 

# force to run provision
vagrant up --provision
# or
vagrant reload --provision

# remove the current vagrant VM
vagrant destroy -f

# shut down and boot up
vagrant halt
vagrant up

# Hibernate
vagrant suspend

# Shut down and up
vagrant reload

# current vagrant vm running status
vagrant status

# Forward ports after Vagrant has started. Forward 7000 on host to 5432 on Vagrant VM
vagrant ssh -- -L 7000:localhost:5432

Connect from Vagrant to Host using Host IP 

# get the default gateway
netstat -rn

# Usually, it is 10.0.2.2

Vagrant Pure SSH 

# Show current vagrant vm ssh config
vagrant ssh-config
# Usually it's 127.0.0.1:2222, notice the port may change as vagrant will auto select port if it's in use

# save to ~/.ssh/config, make it possible to use ssh directly
vagrant ssh-config --host myvagrant >> ~/.ssh/config
ssh myvagrant

21.4. Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :
ENV["LC_ALL"] = "en_US.UTF-8"
# In case host couldn't pass locale into vagrant

hostname = livagrantdev
cpus = 2
memory = 2048

@importmysql = <<SCRIPT
  echo "docker run -it --rm -v /vagrant/mydbdump:/tmp/mydbdump"\
  " --link wordpressdb:wpdb mysql:5.7.9 sh -c"\
  " 'exec mysql"\
  ' -h"$WPDB_PORT_3306_TCP_ADDR" -P"$WPDB_PORT_3306_TCP_PORT"'\
  ' -uroot -p"$WPDB_ENV_MYSQL_ROOT_PASSWORD"'\
  ' wordpress < /tmp/mydbdump/pantheon_db.sql'\
  "'" >> /home/vagrant/import_db.sh
  chmod +x /home/vagrant/import_db.sh
SCRIPT

Vagrant.configure(2) do |config|
  # config.vm
  config.vm.box = username/boxname
  # optional
  # config.vm.box_version = "1.1.0"
  # config.vm.box_url = "http://"
  # hostname, appears when you vagrant ssh - vagrant@yourhostname

  config.vm.hostname = hostname

  # network
  config.vm.network "private_network", ip: "192.168.33.10"
  # VMs in the the private network (VirtualBox) can communicate
  # Give a static IP
  config.vm.network "forwarded_port", guest: 22, host: 2222, id: "ssh"
  config.vm.network "forwarded_port", guest: 80, host: 8080, auto_correct: true
  # On local host (windows), 192.168.33.10:8080

  # Synced folder
  # default /vagrant
  # Create a root folder (readonly) in guest machine to hold Dockerfiles for docker provisioner
  # Create that folder if it doesn't exist in guest machine
  config.vm.synced_folder "./build/docker", "/docker_builds", create: true, mount_options: ["ro"]

  config.vm.provider "virtualbox" do |vb|
       vb.memory = memory
       vb.cpus = cpus
       vb.name = hostname
       vb.gui = true # enable GUI on VirtualBox
       # The name appears on VirtualBox GUI
       #  Can override any settings config.vm, config.ssh, config.winrm and config.vagrant
       end

  # Provision
  # Upload File using SSH user (vagrant)
  config.vm.provision "file", source: "~/.gitconfig", destination ".gitconfig"


  # In line Shell
  config.vm.provision "shell", inline: @importmysql
  # Path to script file Shell
  # config.vm.provision "shell",

  # Provisioner docker - install docker on the VM and pull images!
  config.vm.provision "install_docker", type: "docker" do |d|
      d.pull_images "mysql:5.7.9"
      d.pull_images "wordpress"
      # Need to create a root folder in guest machine to pull Dockerfile
      d.build_image "/docker_builds", args: "-t li-nginx-alpine -f /docker_builds/Dockerfile.li-nginx-alpine"
      # This is the same as
      # docker build -t li-nginx-alpine -f /docker_builds/Dockerfile.li-nginx-alpine /docker_builds/
  end

  # Provisioner docker_compose: install docker_compose on the VM!
  # Require plugin `vagrant-docker-compose`
  config.vm.provision "docker_compose_plugin", type: "docker_compose"

  # config.ssh
  # config.winrm
  # config.vagrant
  end

#Do a loop
  (1..3).each do |i|
      config.vm.define "node-#{i}" do |node|
          node.vm.provision "shell",
              inline: "echo hello from node #{i}"
      end
  end

21.5. Plugin

vagrant-docker-compose
enables a new provisioner Docker Compose
vagrant-vbguest
automatically installs VirtualBox Guest Additions (VBGA) on Vagrant VM
Install
vagrant plugin install vagrant-vbguest
(no term)
To prevent updating VBGA when vagrant up, add config.vbguest.auto_update = false in Vagrantfile. This happens when .box file has older version of vbguest while Vagrant on host has a newer version
vagrant-disksize
resize (upsize only) primary disk for Vagrant instance
  • vagrant plugin install vagrant-disksize

  • Insert this and run vagrant halt and vagrant up 

    vagrant.configure('2') do |config|
config.disksize.size = '50GB'
end
  • While vagrant up, something like this shows up, which means it is resized
    • ==> default: Resized disk: old 10240 MB, req 20480 MB, new 20480 MB
  • vagrant ssh and check if disk size is increased df -h, if not, then "You may need to resize the filesystem from within the guest."
  • Comment out config.disksize.size = '50GB' in Vagrantfile as it's unnecessary to upsize primary disk from now on
Update all installed plugins
vagrant plugin update
List all installed plugins
vagrant plugin list
Uninstall a plugin
vagrant plugin uninstall vagrant-docker-compose
(no term)
After Vagrant core is upgraded, you might need to auto upgrade existing plugins
  • vagrant plugin update vagrant-vbguest
  • vagrant plugin expunge --reinstall
  • vagrant plugin update

21.6. Box

A VirtualBox, .box file, is a compressed box on top of which a VM can run. Box has a namespace username/boxname as in hashicorp/precise64 

# export the running VM to a new box file
vagrant package --output boxfile.box

# decompress a box file to ~/.vagrant.d/boxes
vagrant box add boxname boxfile.box

# boxname is an installed Vagrant box
# destroy current running Vagrant box
vagrant destroy -f
# and
rm Vagrantfile
# Setup another Vagrantfile which uses the newly created box
cofnig.vm.box = "boxname"

# To package an installed Vagrant box (of a specific provider and version)
# into a .box file
vagrant box repackage <boxname> <provider> <version>

# remove a vagrant box
vagrant box remove boxname
# or
vagrant box remove boxname --box-version=0.1.7

# List all installed Vagrant boxes
vagrant box list

# Check if the box used in the current folder is up-to-date
vagrant box outdated

# if it's not outdated and before destroy current Vagrant machine, update the box
vagrant box update
vagrant box update --box boxname
# Destroy the current Vagrant machine and up again.

Vagrant .box should have these qualities:

  • vagrant as a user and map /vagrant to the current folder in host
  • be able to install as the Vagrant provision plugin instructs
    • The ubuntu/xenial64 has a lot of trouble with that and hence it's not a good Vagrant box to start with

21.6.1. bento/ubuntu-16.04

vagrant box add bento/ubuntu-16.04 vagrant box add bento/ubuntu-18.04

21.8. SSH using Putty

  • Download PuttyGen to accept OpenSSH key as vagrant provides
  • PuttyGen > Load > ~/.vagrant.d/ folder and select browse all files, select insecure_public_key
  • PuttyGen > Save public key file as insecure_public_key.ppk and save it to the same folder. Close PuttyGen and go back to Putty
  • Putty > Connection > SSH > Auth, browse and select that ppk file.
  • Connect using Putty!
  • Specify the username for login: Putty > Connection > Data > Auto-login username
  • ~/.vagrant.d/ is the global. Different box has ./.vagrant/ folder. Run vagrant ssh-config to identify the $HOSTNAME $PORT $LOGINNAME

Convert the Identityfile to .ppk with SSH-2-RSA with 2048 bits and save it to the same folder.

22. Apache

22.1. Basics

Run this to get Apache version, root and config file httpd -V or apache2 -v in Ubuntu

22.2. Config loading

  • apt-get update; apt-get install apache2
  • Refer to image:php:apache to get configs
  • apache2ctl -V or httpd -V to look for HTTPD_ROOT and SERVER_CONFIG_FILE to find the apache:config:main file

  • apache2ctl -S 

    VirtualHost configuration:
*:80                   172.17.0.7 (/etc/apache2/sites-enabled/000-default.conf:1)

# run settings followed

ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex default: dir="/var/lock/apache2" mechanism=fcntl
Mutex mpm-accept: using_defaults
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33
Group: name="www-data" id=33

22.2.1. Loaded Modules

Show all loaded modules
apache2ctl -M
(no term)
Sample loaded modules from image:php:apache
(no term)
You can also ls -al /etc/apache2/mods-enabled

autoindex deflate expires filter mime rewrite setenvif mpm_prefork

22.2.2. sites-enabled VirtualHost

Sites
etc/apache2/sites-enabled
(no term)
Sample for image:php:apache
(no term)
Use Listen directive to listen ports other than the default 80 and 443. Refer to apache:ports.conf
<VirtualHost *:80>
        # The ServerName directive sets the request scheme, hostname and port that
        # the server uses to identify itself. This is used when creating
        # redirection URLs. In the context of virtual hosts, the ServerName
        # specifies what hostname must appear in the request's Host: header to
        # match this virtual host. For the default virtual host (this file) this
        # value is not decisive as it is used as a last resort host regardless.
        # However, you must set it for any further virtual host explicitly.
        #ServerName www.example.com

        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html

# You may add Directory here
# <Directory /var/www/html>
#   Options Indexes FollowSymLinks MultiViews
#   AllowOverride All
#   Order allow,deny
#   allow from all
# </Directory>

        # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
        # error, crit, alert, emerg.
        # It is also possible to configure the loglevel for particular
        # modules, e.g.
        #LogLevel info ssl:warn

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        # For most configuration files from conf-available/, which are
        # enabled or disabled at a global level, it is possible to
        # include a line for only one particular virtual host. For example the
        # following line enables the CGI configuration for this host only
        # after it has been globally disabled with "a2disconf".
        #Include conf-available/serve-cgi-bin.conf
</VirtualHost>

22.2.3. Main Config files

  • Main config file (one of the following)
    • /etc/apache2/apache2.conf
    • /etc/httpd/httpd.conf
    • /usr/local/apache/conf/httpd.conf
    • Run apache:config to get Apache root directory to find config location

  • *.conf files loading sequence 

    # include module config
IncludeOptional mods-enabled/*.load
IncludeOptional mods-enabled/*.conf

# include list of ports to listen on
Include ports.conf

# main config file config

# generic
IncludeOptional conf-enabled/*.conf
# virutal host config
IncludeOptional sites-enabled/*.conf
    • ports.conf
      • Default only listen 80 and 443
      • Listen 8081
  • Include https://github.com/h5bp/server-configs-apache/blob/master/dist/.htaccess inside <Directory> section

22.2.4. Restart

  • Check config before restart. Wrong config (e.g. SSL) may prevent Apache from starting
    • apache2ctl configtest
    • apachectl configtest
    • httpd -t
  • Reload
    sudo systemctl reload apache2.service
    Ubuntu
    /etc/init.d/apache2 reload
    for apache that is initialized using command e.g. php apache docker container
    apache2ctl graceful
    to stop and start httpd daemon
  • sudo systemctl restart apache2
    • apache2ctl restart
    • service apache2 restart
    • /etc/init.d/apache2 restart
    • service httpd restart
  • On Ubuntu, use systemctl status apache2.service

22.2.5. a2ensite, a2dissite

  • /etc/apache2/sites-available/000-client1-site1.conf
    • Should have <VirtualHost> block
  • sudo a2ensite 000-client1-site1
    • create a symlink in /etc/apache2/sites-enabled/
  • sudo a2dissite 000-client1-site1
    • remove a symlink in /etc/apache2/sites-enabled
  • apache:reload

22.3. Modules

  • Available modules
    • cat /etc/apache2/mods-available
  • Enabled modules
    • cat /etc/apache2/mods-enabled
    • apache2ctl -M
    • apache2ctl -M | grep headers_module
  • Terms used to describe modules
    • Status
      Base
      they're compiled and loaded into the server by default

  • Command line enable modules 

    a2enmod remoteip
# some modules need to load config. If it doesn't specify, then the module does not need a2enconf
a2enconf remoteip

a2enmod rewrite
  • 22.2.4

22.3.1. remoteip

https://github.com/wichon/docker-php-maxmind-geoip/blob/master/Dockerfile

When Nginx proxy Apache, Apache's remote_addr is the Nginx's IP (gateway of the current docker container)

Create remoteip.conf file under /etc/apache2/conf-available/remoteip.conf 

RemoteIPHeader X-Forwarded-For
RemoteIPProxiesHeader X-Forwarded-By

a2enmod remoteip && a2enconf remoteip
# then apache:restart

22.3.2. mod_setenvif

<IfModule mod_setenvif.c>
    <IfModule mod_headers.c>
    <FilesMatch "\.(bmp|cur|gif|ico|jpe?g|png|svgz?|webp)$">
        SetEnvIf Origin ":" IS_CORS
        Header set Access-Control-Allow-Origin "*" env=IS_CORS
    </FilesMatch>
    </IfModule>
</IfModule>

22.3.3. mod_headers apache:mod_headers

<IfModule mod_headers.c>
    Header set Access-Control-Allow-Origin "*"
</IfModule>

<IfModule mod_headers.c>
    <FilesMatch "\.(eot|otf|tt[cf]|woff2?)$">
    Header set Access-Control-Allow-Origin "*"
    </FilesMatch>
</IfModule>
22.3.3.1. apache:mod_headers:cache

Refer to header:cache-control, 1.37 

<filesMatch "\.(html|htm|js|css)$">
  FileETag None
  <ifModule mod_headers.c>
     Header unset ETag
     Header set Cache-Control "max-age=0, no-cache, no-store, must-revalidate"
     Header set Pragma "no-cache"
     Header set Expires "Wed, 11 Jan 1984 05:00:00 GMT"
  </ifModule>
</filesMatch>
22.3.3.2. apache:mod_headers:header
Syntax
Header [condition] add|append|echo|edit|edit*|merge|set|setifempty|unset|note header [[expr=]value [replacement] [early|env=[!]varname|expr=expression]]
(no term)
All apache:directive context
(no term)
Eq. to nginx:d:add_header
(no term)
condition
  • always
(no term)
env
env=varname
applies if varname is defined
env=!varname
applies if varname is not defined
<IfModule mod_headers.c>
    Header unset X-Powered-By
</IfModule>

Add flags for cookies 

Header edit Set-Cookie ^(.*)$ $1;HttpOnly;Secure

22.3.4. mod_include

22.3.5. mod_rewrite

22.3.5.1. RewriteEngine
Syntax
RewriteEngine on|off
Default
RewriteEngine off
Context
server config, virtual host, directory, .htaccess
(no term)
Usage
  • In a particular context, disable all rewrite rules rather than commenting out all apache:rewriterule
  • Each virtual host should have standalone RewriteEngine on if the virtual hosts need to rewrite
<IfModule mod_rewrite.c>
RewriteEngine on
# ...
</IfModule>
22.3.5.2. RewriteBase

Syntax: RewriteBase [URL-path] specifies the URL prefix to be used for per-directory (htaccess) RewriteRule directives that substitute a relative path

Request GET /somepath/localpath/pathinfo 

RewriteBase "/somepath"
RewriteRule ^localpath(.*) otherpath$1
# /somepath/otherpath/pathinfo
RewriteRule ^localpath(.*) /otherpath$1
# /otherpath/pathinfo
RewriteRule ^localpath(.*) http://thishost/otherpath$1
# /otherpath/pathinfo
22.3.5.3. RewriteCond
Syntax
RewriteCond [TestString] [CondPattern] [flags]
(no term)
CondPattern can be
  • regex
  • is, not a directory
  • is, not a file
RewriteCond %{REMOTE_ADDR} ^1\.2\.3\.4$

RewriteCond %{HTTP_HOST} ^xyz\-40\.ca$ [OR]
RewriteCond %{HTTP_HOST} ^www\.xyz\-40\.ca$
RewriteRule ^(.*)$ "https\:\/\/www\.xyz\.com\/$1" [R=302,L]
  1. Server-Variables
    1. ${HTTP_COOKIE}
      RewriteCond %{HTTP_COOKIE}     its=([^;]+) 
RewriteCond %1                 ^me$
RewriteRule ......

RewriteCond %{HTTP_COOKIE}     its=([^;]+) 
RewriteCond %{unescape:%1}     ^me$
RewriteRule ......
    2. %{TIME_*}
      <IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteBase /
  RewriteRule ^dealdays/?(.*)$ /DealDays/$1 [R=302,L]
  RewriteCond %{TIME_YEAR}%{TIME_MON}%{TIME_DAY}%{TIME_HOUR} >2017110521
  RewriteRule ^DealDays/?(.*)$ / [R=302,L]
</IfModule>
  2. RewriteCond Flags
    NC
    no case
    NV
    no vary
    (no term)
    OR
    RewriteCond "%{REMOTE_HOST}"  "^host1"  [OR]
RewriteCond "%{REMOTE_HOST}"  "^host2"  [OR]
RewriteCond "%{REMOTE_HOST}"  "^host3"
RewriteRule ...some special stuff for any of these hosts...
22.3.5.4. RewriteRule
Syntax
RewriteRule [Pattern] [Substitution] [flags]
Context
server config, virtual host, directory, .htaccess
Require
apache:rewriteengine on
(no term)
Similar to 23.8
(no term)
Pattern

  • Match what?

    Virtual Host context
    '/app1/index.html'
    • The part of URL after the hostname and port and before the query string
    • Decoded
      Per-directory context (Directory and .htaccess)
      'app1/index.html'
    • A relative path could be one of the following
      • current directory's .htaccess
      • <Directory "/path/to/app/">)
      • apache:rewritebase
      • Previous RewriteRule substition
      • apache:documentroot or apache:alias
  • regex
    • | ^ $ ( ) [ ] . * + ?
    • \d
    • a-Z, 0-9 and underscore
(no term)
Substitution
"-" or -
the requested URI is not modified
(no term)
If it doesn't have a leading slash, the end URL has the current directory's .htaccess or apache:rewritebase
(no term)
Use double quotes
  • Pattern and Substitution can be wrapped with double quotes
  • When # is used in Pattern or Substituion. If it is in Substition, 22.3.5.5 NE is required
22.3.5.5. RewriteRule Flags
  • https://httpd.apache.org/docs/2.4/rewrite/flags.html
  • Use comma to separate flags NOT spaces!
  • return 403 and L is implied e.g. RewriteRule "\.exe" "-" [F]
  • Last
  • no case
  • no escape. By default, special characters in Substituion wrapped in quotes will be escaped e.g. & ? #
  • set MIME type with which the resulting response will be sent
  • set environment variable e.g. [E=VAR:VAL] [E=VAR] (set to empty) [E=!VAR] (unset a previously set env var named VAR)
  • handle request by mod_proxy
  • Default behavior of RewriteRule is to discard the existing query string, and replace it with the newly generated one. Use it to cause the query string to be combined
    • e.g. RewriteRule "/pages/(.+)" "/page.php?page=$1" [QSA]
      • /pages/123?one=two will be mapped to /page.php?page=123&one=two
    • There's no QSA equivalent in Nginx because by default Nginx appends query parameters
  • force an external redirect, optionally with the specified HTTP status code
22.3.5.6. Rewrite a path to sub directory with .htaccess
  • URL /inventory/* rewrite to /my-app/public/*
    • ./my-app/public/index.php is where the app starts
    • ./my-app/app/*.* are the app codes
    • ./my-app/public/theme/*.* and ./my-app/public/uploads/*.* have the assets

  • ./.htaccess 

    <IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^inventory/?$ /my-app/public/index.php [L]
RewriteRule ^inventory/(.*)$ /my-app/public/$1 [L]
</IfModule>

  • ./my-app/public/.htaccess 

    <IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /my-app/public/
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /my-app/public/index.php [L]
</IfModule>
22.3.5.7. WordPress Example
  • RewriteRule with redirect flag [R] is to redirect URL. Without it, RedirectRule maintains the URL on LHS in the address bar.

Redirect and RedirectMatch is to redirect Redirect means the URL in the address bar will change

RewriteRule without proxy flag [P] is to maintain the URL on LHS as $_SERVER['REQUEST_URI'] in RewriteRule RewriteRule with [P] is to change the URL to the one on RHS as the new $_SERVER['REQUEST_URI'] in RewriteRule and pass it to the proxy [P] requires mod_proxy enabled

The only way to modify $_SERVER['REQUEST_URI'] is to do a redirect or with the [P] flag.

RewriteRule is relative to the directory that the current .htaccess is in for both RHS and LHS. RewriteBase only affects the dest of RewriteRule (RHS) only if dest is a relative path

Redirect /dealdays to /DealDays 

# custom RewriteRule goes before the wordpress block
<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteBase /
  RewriteRule ^dealdays/?(.*)$ /DealDays/$1 [R=302,L]
  RewriteRule ^abc/?(.*)$ /subdir/abc.php [L]
</IfModule>
# custom RewriteRule end

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

# default wordpress htaccess block is to redirect to index.php if it's not a file and not a directory
# Rules below the block has to be either a file or an existing directory

RewriteRule ^dealdays/(.*)$ /DealDays/$1 [R=302,L]
# If redirect to another url, use
# Redirect is prefix match and append additional path info beyond the matched URL-path to the new/target URL.
# a.com/dealdays2 to b.com/dealdays2, a.com/dealdays2/foo.txt to b.com/dealdays2/foo.txt, a.com/dealdays2/xyz to b.com/dealdays2/xyz
# Redirect syntax :: Redirect [status] [URL-path] URL
# [URL-path] is case-sensitive beginning with slash. relative path is not allowed
# URL may be either an absolute URL beginning with scheme and hostname or a URL-path beginning with a slash
Redirect 301 /dealdays2 http://b.com/dealdays2

# Redirect without appending additional path info
# RedirectMatch is like Redirect but regex
# syntax :: RedirectMatch [status] regex URL
RedirectMatch 302 ^/dealdays2/?$ http://b.com/dealdays2

# case insensitive just add (?i) in front of the pattern
RedirectMatch 302 (?i)^dealdays2/?$  /dealdays/

Redirect root domain only but not other URI. e.g. mysite.com or www.mysite.com redirect to abc.com but not mysite.com/xyz to abc.com/xyz 

RewriteEngine on
RewriteCond %{HTTP_HOST} ^(www\.)?mysite\.com [NC]
# no www :: RewriteCond %{HTTP_HOST} ^mysite\.com [NC]

RewriteCond %{REQUEST_URI} ^/$
# Rewriterule ^(.*)$ http://abc.com/ [L,R=302]
Rewriterule ^$ http://abc.com/ [L,R=302]

Redirect www.abc.com to abc.com 

RewriteCond %{HTTP_HOST}  ^www.abc.com [NC]
RewriteRule ^(.*) http://abc.com/$1 [L,R=302]

22.3.6. mod_autoindex

If the URI matches a directory but doesn't specify an index file e.g. index.html, by default Apache returns the index of that directory. Don't do that! 

<IfModule mod_autoindex.c>
    Options -Indexes
</IfModule>

22.3.7. mod_mime

22.3.7.1. AddType TypesConfig apache:mod_mime:addtype
  • Context: all
AddType media-type extension [extension] ...

AddType image/webp .webp

AddType font/woff2 woff2

AddType image/jpeg jpeg jpg jpe

/etc/apache2/mods-available/mime.conf loads MIME types using TypesConfig /etc/mime.types 

echo "font/woff2 woff2" >> /etc/mime.types
22.3.7.2. AddEncoding

AddEncoding encoding extension [extension] … extension may or may not have leading dot. This doesn't encode the file but rather sets the response header Content-Type. See apache:mod_filter:addoutputfilterbytype 

<IfModule mod_mime.c>
  AddEncoding gzip svgz
</IfModule>

22.3.8. mod_dir

22.3.8.1. DirectoryIndex
DirectoryIndex index.html index.txt  /cgi-bin/index.pl

# http://example.com/docs/index.html
# could cause the CGI script /cgi-bin/index.pl to be executed if neither index.html or index.txt existed in a directory.

22.3.9. mod_filter

22.3.9.1. AddOutputFilterByType apache:mod_filter:addoutputfilterbytype

AddOutputFilterByType filter[;filter…] media-type [media-type] … Default deflate content types :: /etc/apache2/mods-available/deflate.conf 

<IfModule mod_filter.c>
AddOutputFilterByType DEFLATE "application/atom+xml" \
                              "application/javascript" \
                              "application/json" \
                              ...
                              "image/svg+xml" \
                              ...etc.
</IfModule>

22.4. Directive Context apache:directive context

server config
This means that the directive may be used in the server configuration files (e.g., httpd.conf), but not within any <VirtualHost> or <Directory> containers
  • Not allowed in .htaccess files at all.
virtual host
This context means that the directive may appear inside <VirtualHost> containers in the server configuration files
directory
A directive marked as being valid in this context may be used inside <Directory>, <Location>, <Files>, <If>, and <Proxy> containers in the server configuration files, subject to the restrictions outlined in Configuration Sections
.htaccess
If a directive is valid in this context, it means that it can appear inside per-directory .htaccess files. It may not be processed, though depending upon the overrides currently active

22.5. Core Directives

These can be used in all apache:directive context. However, some directive setting needs to happen in a different context. https://httpd.apache.org/docs/2.4/mod/core.html 

ErrorDocument 404 /404.html

# This setting prevents Apache from returning a 404 error as the result
# of a rewrite when the directory with the same name does not exist.
Options -MultiViews

22.5.1. AllowOverride

AllowOverride All|None|directive-type [directive-type] … context: only directory default: AllowOverride None (2.3.9 and later), AllowOverride All (2.3.8 and earlier)

Which directives declared in .htaccess can override earlier configuration directives.

None :: .htaccess files are completely ignored.

22.5.2. FilesMatch

Force to download a PDF file. The path doesn't matter 

<FilesMatch "aPDFfile.pdf">
  ForceType application/octet-stream
  Header add Content-Disposition "attachment"
</filesMatch>

22.5.3. ServerSignature

  • ServerSignature On|Off|EMail
  • off
  • server config, virtual host, directory, .htaccess

22.5.4. LogLevel

Syntax
LogLevel [module:]level [module:level] ...
Default
LogLevel warn
Context
server config, virtual host, directory
(no term)
https://httpd.apache.org/docs/2.4/mod/core.html#loglevel
(no term)
Put it inside VirtualHost e.g. in /etc/apache2/sites-available/*.conf
# for all modules use alert level, but for rewrite module, level is trace6, which is extremely detailed
LogLevel alert rewrite:trace6

# sometimes, mod_rewrite.c:trace6 is required

# For < 2.4 use this for Rewrite log
RewriteEngine On
RewriteLog "/var/log/apache2/rewrite.log"
RewriteLogLevel 3

trace1 to trace8 (most detailed)

22.5.5. Include

Syntax
Include file-path|directory-path|wildcard
Context
server config, virtual host, directory
Relative path
relative to ServerRoot e.g. /etc/apache2
Include conf/ssl.conf # /etc/apache2/conf/ssl.conf
Include conf/vhosts/*.conf
Include /path/to/conf/*.conf # fail with an error if there is no file matched

22.6. LogFormat

LogFormat
https://httpd.apache.org/docs/2.4/mod/mod_log_config.html#logformat
  • LogFormat associate a format string to a nickname (e.g. common), nickname is optional
  • CustomLog specifies the log file location for that nickname
Context
server config, virtual host
Default
LogFormat "%h %l %u %t \"%r\" %>s %b"
Custom Log Formats
https://httpd.apache.org/docs/2.4/mod/mod_log_config.html#formats
LogFormat "%h %l %u %t \"%r\" %>s %b" common
CustomLog logs/access_log common

Combined more fields 

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined
CustomLog log/access_log combined

# 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326
# after combined
# 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08 [en] (Win98; I ;Nav)"

2326 :: %b :: size of the object returned to the client, not including the response headers. If no content was returned to the client, this value will be "-". To log "0" for no content, use %B instead.

22.7. Basic authentication

htpasswd creates auth file and it needs this package. htpasswd doesn't need to be installed on server. 

apt-get install apache2-utils

# use -c to create and sammy is the username, supply password later
sudo htpasswd -c /mywebsite/.htpasswd-dev sammy

# later add more user without -c
sudo htpasswd -c /mywebsite/.htpasswd-dev another_user

.htpasswd file content 

Output
sammy:$apr1$.0CAabqX$rb8lueIORA/p8UzGPYtGs/
another_user:$apr1$fqH7UG8a$SrUxurp/Atfq6j7GL/VEC1

MD5 \(apr1\).0CAabqX$rb8lueIORA/p8UzGPYtGs/ 32bit salt is .0CAabqX MD5 result is rb8lueIORA/p8UzGPYtGs/

22.7.1. Option 1: control access within Virutal Host Definition

/etc/apache2/sites-enabled/000-default.conf 

<VirtualHost *:80>
  ServerAdmin webmaster@localhost
  DocumentRoot /var/www/html
  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined

# add this
  <Directory "/var/www/html">
    AuthType Basic
    AuthName "Restricted Content"
    AuthUserFile /etc/apache2/.htpasswd
    Require valid-user
  </Directory>
# add this.

</VirtualHost>

22.7.2. Option 2: control access with .htaccess

Main config file :: etc/apache2/apache2.conf Allow .htaccess to override config For /var/www, change AllowOverride None to All 

. . .
<Directory /var/www/>
  Options Indexes FollowSymLinks
  AllowOverride All
  Require all granted
</Directory>
. . .

var/www/mywebsite.htaccess, restart apache 

AuthType Basic
AuthName "restricted area"
AuthUserFile /var/www/__secure/.htpasswd-dev
require valid-user

22.8. Production Setup apache:production

This guide is for httpd. Optimize Webhost Setup nano /etc/httpd/conf/httpd.conf 

Timeout 30
KeepAlive on
MaxKeepAliveRequests 50
KeepAliveTimeout 15
#Changes to prefork module
<IfModule prefork.c>
  StartServers 3
  MinSpareServers 2
  MaxSpareServers 5
  ServerLimit 256
  MaxClients 10
  MaxRequestsPerChild 1000
</IfModule>

Search AllowOverrride and set it to All for 

<Directory />
  Options FollowSymLinks
  AllowOverride All
</Directory>

And 

# AllowOverride controls what directives may be placed in .htaccess files.
# It can be "All", "None", or any combination of the keywords:
# Options FileInfo AuthConfig Limit
#
AllowOverride All

Restart Apache service httpd restart

22.9. Force redirect from non www to www

<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{HTTP_HOST} !^www\. [NC]
    RewriteRule ^(.*)$ http://www.%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
</IfModule>

Redirect from www.example.com to example.com 

<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{HTTP_HOST} ^www\.(.+)$ [NC]
    RewriteRule ^ %{ENV:PROTO}://%1%{REQUEST_URI} [R=301,L]
</IfModule>

22.10. Force redirect to https .htaccess

Diffrent hositng companies have different rule

In general 

<IfModule mod_rewrite.c>
RewriteEngine On
# RewriteBase is optional but some hosting requires it
RewriteBase /
RewriteCond %{HTTPS} !=on
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
</IfModule>

InMotion https://www.inmotionhosting.com/support/website/ssl/how-to-force-https-using-the-htaccess-file 

<IfModule mod_rewrite.c>
    RewriteCond %{REQUEST_URI} !^/[0-9]+\..+\.cpaneldcv$
    RewriteCond %{REQUEST_URI} !^/\.well-known/pki-validation/[A-F0-9]{32}\.txt(?:\ Comodo\ DCV)?$
    RewriteEngine On 
    RewriteCond %{SERVER_PORT} 80 
    RewriteRule ^(.*)$ https://www.example.com/$1 [R=301,L]
</IfModule>

22.11. Restrict access to a file or sub folder

var/www.htaccess blocks access to folders cs-devops and .git and file .gitignore. This way log gets 403 and apache doesn't need to be restarted 

RewriteRule ^(cs-devops/|\.git/|\.gitignore|docker-compose\.yml|Makefile|README\.md|error_log|wp-config-li-[^.]+\.php) - [F,NC]

22.12. WebP

<IfModule mod_rewrite.c>

  RewriteEngine On

  # Check if browser support WebP images
  RewriteCond %{HTTP_ACCEPT} image/webp

  # Check if WebP replacement image exists
  RewriteCond %{DOCUMENT_ROOT}/$1.webp -f

  # Serve WebP image instead
  RewriteRule (.+)\.(jpe?g|png)$ $1.webp [T=image/webp,E=accept:1]

</IfModule>

<IfModule mod_headers.c>

    Header append Vary Accept env=REDIRECT_accept

</IfModule>

AddType  image/webp .webp

22.13. SSL apache:ssl

  • Docs from DigiCert
  • Find config file that handles SSL
    • grep -i -r "SSLCertificateFile" /etc/httpd/ or
    • grep -i -r "SSLCertificateFile" /etc/apache2/
  • sudo a2enmod ssl
  • The SSLCertificateFile may also contain intermediate certificates, DH parameters and EC curve.

  • Add SSL* directives 

    # May need for port 80 as well
<VirtualHost 192.168.0.1:80>
DocumentRoot /var/www/html2
ServerName www.yourdomain.com
    SSLEngine on
    SSLCertificateFile /path/to/your_domain_name.crt
    SSLCertificateKeyFile /path/to/your_private.key
    SSLCertificateChainFile /path/to/DigiCertCA.crt
</VirtualHost>

<VirtualHost 192.168.0.1:443>
DocumentRoot /var/www/html2
ServerName www.yourdomain.com
    SSLEngine on
    SSLCertificateFile /path/to/your_domain_name.crt
    SSLCertificateKeyFile /path/to/your_private.key
    SSLCertificateChainFile /path/to/DigiCertCA.crt
</VirtualHost>
  • Check before apache:restart

22.14. Redirect to HTTPS apache:https

RewriteEngine On 
RewriteCond %{SERVER_PORT} 80 
# or
# RewriteCond %{HTTPS} off

# test one page
# RewriteCond %{REQUEST_URI} ^/page-path-name/$

RewriteRule ^(.*)$ https://www.example.com/$1 [R,L]
# or
# RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R,L]

A specific domain 

RewriteEngine On 
RewriteCond %{HTTP_HOST} ^example\.com [NC]
RewriteCond %{SERVER_PORT} 80 
RewriteRule ^(.*)$ https://www.example.com/$1 [R,L]

Force SSL on a specific folder. Place this .htaccess in that folder 

RewriteEngine On 
RewriteCond %{SERVER_PORT} 80 
RewriteCond %{REQUEST_URI} folder 
RewriteRule ^(.*)$ https://www.example.com/folder/$1 [R,L]

23. Nginx

23.1. Install on Ubuntu 16.04

# Ubuntu 16.04
sudo apt-get update
sudo apt-get install nginx

nginx:ufw

23.1.1. Config Firewall nginx:ufw

This is optional if firewall ufw doesn't exist.

Nginx registers itself as a service with firewall ufw 

sudo ufw app list

# Available applications:
#  Nginx Full
#  Nginx HTTP
#  Nginx HTTPS
#  OpenSSH
Nginx Full
opens both port 80 (normal, unencrypted web traffic) and port 443 (TLS/SSL encrypted traffic)
Nginx HTTP
opens only port 80 (normal, unencrypted web traffic)
Nginx HTTPS
opens only port 443 (TLS/SSL encrypted traffic)

Enable Nginx HTTP 

sudo ufw allow 'Nginx HTTP'
sudo ufw status

# output
# Status: active

# To                         Action      From
# --                         ------      ----
# OpenSSH                    ALLOW       Anywhere                  
# Nginx HTTP                 ALLOW       Anywhere                  
# OpenSSH (v6)               ALLOW       Anywhere (v6)             
# Nginx HTTP (v6)            ALLOW       Anywhere (v6)

Enable HTTPS, allow Full and delete HTTP 

sudo ufw allow 'Nginx Full'
sudo ufw delete allow 'Nginx HTTP'

23.1.2. Service Check, reload, restart, Check syntax

  • Check syntax for config before restart :: sudo nginx -t
  • Nginx can be run with daemond off by nginx -g 'daemon off;'
    • Then use /etc/init.d/nginx instead of systemctl nor service
  • systemctl stop/start/restart/reload/disable/enable nginx
    status
    systemctl status nginx
    restart
    service nginx restart
    • or sudo /etc/init.d/nginx restart
    (no term)
    reload config files only
    • sudo nginx -s reload
    • or sudo /etc/init.d/nginx reload
    • restart drops connection, use reload to not drop connection
    disable
    nginx service, default the nginx service is to start automatically

  • nginx -v Output 

    ● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2016-04-18 16:14:00 EDT; 4min 2s ago
 Main PID: 12857 (nginx)
   CGroup: /system.slice/nginx.service
	   ├─12857 nginx: master process /usr/sbin/nginx -g daemon on; master_process on
	   └─12858 nginx: worker process

23.2. /etc/nginx /var/log/nginx

23.2.1. etc/nginx/sites-available

The directory where per-site "server blocks" can be stored. Nginx will not use config files in this directory unless they are linked to the sites-enabled directory.

There should be a default file you can use. /etc/nginx/sites-available/default 

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    root /var/www/html;
    index index.html index.htm index.nginx-debian.html;

    server_name _;

    location / {
        try_files $uri $uri/ =404;
    }
}

ATTENTION! Only one server block can have default_server Search if there is any file has default_server 

grep -R default_server /etc/nginx/sites-enabled/

sudo cp /etc/nginx/sites-available/default /etc/nginx/sites-available/example.com
sudo nano /etc/nginx/sites-available/example.com

/etc/nginx/sites-available/example.com 

server {
        listen 80;
        listen [::]:80;

        root /var/www/example.com/html;
        index index.html index.htm index.nginx-debian.html;

        server_name example.com www.example.com;

        location / {
                try_files $uri $uri/ =404;
        }
}

Enable the server block and restart Nginx

sudo ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/example.com

In order to avoid a possible hash bucket memory problem that can arise from adding additional server names, we will go ahead and adjust a single value within our /etc/nginx/nginx.conf file. Open the file now: 

sudo nano /etc/nginx/nginx.conf

Within the file, find the server_names_hash_bucket_size directive. Remove the # symbol to uncomment the line: 

http {
    . . .

    server_names_hash_bucket_size 64;

    . . .
}

Check syntax 

sudo nginx -t
sudo systemctl restart nginx

23.2.2. etc/nginx/sites-enabled

The directory where enabled per-site "server blocks" are stored.

Niginx on Debian or Ubuntu include /etc/nginx/sites-enabled/* while other installation have server blocks set up /etc/nginx/conf.d/*.conf

23.2.3. etc/nginx/snippets

This directory contains config fragments that can be included elsewhere in the Nginx config.

23.2.4. /var/log/nginx/access.log

23.2.5. /var/log/nginx/error.log

23.3. Installed modules

# To check if a module (http_stub_status_module) is installed. It returns the module name if it's installed

nginx -V 2>&1 | grep -o with-http_stub_status_module

23.3.1. Stub status module

Create this location directive under any server directive to see the status info for the whole Nginx status for all servers and connections 

server {
  listen 81;
  server_name localhost;

  access_log off;
  allow 127.0.0.1;
  deny all;

  location /nginx_status {
    # Choose your status module

    # freely available with open source NGINX
    stub_status;

    # for open source NGINX < version 1.7.5
    # stub_status on;

    # available only with NGINX Plus
    # status;

    # ensures the version information can be retrieved
    server_tokens on;
  }
}

23.4. Config

  • Check syntax and restart nginx:restart

  • Check a setting 

    cd /etc/nginx/
grep -r 'client_max_body_size' .

23.4.1. Main global config /etc/nginx/nginx.conf

user www-data;
worker_processes 4;
pid /run/nginx.pid;

events {
  ...
}

http {
  sendfile off;
  tcp_nopush on;
  tcp_nodelay on;
  keepalive_timeout 65;
  types_hash_max_size 2048;
  include /etc/nginx/mime.types;
  default_type application/octet-stream;

  access_log /var/log/nginx/access.log ;
  error_log /var/log/nginx/error.log;

  include /etc/nginx/conf.d/*.conf;
}

23.4.2. Define server(s) in /etc/nginx/conf.d/default.conf

# comment starts here.
# You should always add a server to catch any requests without "Host" header
server {
  listen 80;
  server_name ""; # this line can be omitted if version > 0.8.48
  return 444;
}

# another server
server {

}

23.5. server Block nginx:b:server

Include directives
listen, server_name, root, index
Include blocks
location
server {
  listen 80 default_server;
# or without any default_server in listen, the first server with listen is the default server
# listen 80;

# listen actually is a combo of ip and port 192.168.1.1:80
# nginx first identify which ip:port has the incoming request, then try to match the server_name from the request
# if no server_name matches, then it goes to the default_server for ip:port
# If no ip is set in ip:port, then 0.0.0.0 is used
# If no port is set in ip:port, then 80 is used
# If no listen is set, then 0.0.0.0:80 is used
# If only ip is specified (no port), it has a higher specificity than only port is specified.
# 0.0.0.0 means all ipv4 addresses
# listen [::]:80 ipv6only=on; # means to listen all ipv6 address :::80

  server_name example.org www.example.org;

# server_name :: can contain wildcards that either at start or end.
# *.example.org matches www.example.org and www.sub.exmple.org as well.
# nginx first searches for the exact match of a prefix given by literal strings (no regex, no wildcard)
# If not found, then checks for leading wildcard first then trailing wildcard (no regex)
# In each case if there're multiple matches, the longest match will be used
# Then nginx checks server_name given by regex in the order listed
# The first matching regex stops the search and nginx will use that server_name
# If no regex is matched, then uses the most specific prefix found earlier

# server_name regex always starts with ~. Recommended to always start with ^ and end with $
# e.g. server_name ~^www\d+\.example\.net$;
# regex containing { and/or } should be quoted entirely
# e.g. server_name "~^(?<name>\w\d{1,3}+)\.example\.net$"
# named capture :: later $name can be used
# always use named capture as $1 will be passed down to any sub directives and server_name is usually
# one of the top directives

# Special server_name :: "" you can do this
# server_name example.org "";
# This matches any server name
# server_name _;

  index index.cfm index.html index.htm;
  root /var/www;

# root combines with uri to get a file. e.g. /var/www/uripart1/uripart2
# If the result is a directory and doesn't have an index file (e.g. index.cfm set in index)
# Then it will return 403 forbidden
# Turn on autoindex will list files in directory
# autoindex on; 

# include any common config files. /etc/nginx/common/*.conf
  include common/common.conf
}

23.6. location block

23.6.1. Basics

# location syntax
# location optional_modifier location_match {...}
# location only matches URI without URL parameters

# modifier =
location = /page1 {
    # exact match
    # if there is a match, return this immediately
}

# modifier ^~, non regex match
location ^~ /site {
    # This is a prefix match: /site/*
    # If this match (with ^~) is the longest prefix match, return immediately
}

# no modifier, the location_match part is a prefix
location /site {
    # This is also a prefix match: handles /site, /site/*, /site/page1/index.html
    # If this match (without ^~) is the longest prefix match, 
    # save for the moment and move on to regex matches
}

# modifier ~ or ~*, regex match
location ~ \.(jpe?g|png|gif|ico)$ {
    # ~  :: case-sensitive,
    # ~* :: is case-insensitve
    # By now, if there is a prefix match (also longest prefix match), it's already stored in memory
    # These regex matches are checked by the listed order
    # Regex match searching terminates on the first match, and return immediately
    # If no regex match, then return the previous found longest prefix match

    # So if there's a regex match, then the regex match will be used

    # Also notice, if there're any regex matches that are within the longest prefix match, 
    # those will be evaluated, in listed order, before any of the other regex matches.
    # Regex match inside prefix match > Stand alone regex match even though there's prefix match
}

location ~ ^/(digital-archive|digital-edition)/?$ {
    # $ can be used to mark the end
    return 302 http://www.target.ca/pub/abc/;
}

location ~ ^/(subscribe|contacts|advertise|privacy-policy)/?$ {
    # matched: /subscribe /subscribe/
    # not matched: /subscribe/abc
    rewrite (?i)^/([^/]+)$ https://www.target.ca/$1 last;
}

location ~ ^/(segment|keyword|category|tag)/[^/]+ {
    # matched: /tag/abc
    # not matched: /tag/ /tag 
    rewrite (?i)/(.*)$ https://www.target.ca/$1 last;
}

location ~ ^/[^/]+/?$ {
    # matches https://source.ca/abc and /abc/
    # but not https://source.ca/abc/xyz
}

# one location can "jump" to another location
root /var/www/main;
location / {
    rewrite ^/rewriteme/(.*)$ /$1 last;
    try_files $uri $uri.html $uri/ /fallback/index.html;
}

location /fallback {
    root /var/www/another;
}

# /rewriteme/hello > /hello > /hello, /hello.html, /hello/, /fallback/index.html

# one location can "jump" to another location :: custom 404
root /var/www/main;
location / {
    error_page 404 /another/whoops.html;
}

location /another {
    root /var/www;
}

23.6.2. Subdirectory as a React app

server {
    gzip on;
    gzip_types text/css application/x-javascript application/javascript text/javascript text/x-javascript;
    gzip_vary on;

    listen 80;

    access_log /var/log/nginx/access.log gslog if=$loggable;

    server_name localhost;
    root /var/www/;

    client_max_body_size 120M;

    location / {
        index index.php;
        autoindex off;
    }

    location /path/to/react/app {
        index index.php;
        autoindex off;
        try_files $uri $uri/ /path/to/react/app/index.php;
    }

    location ~ \.php$ {
        fastcgi_pass 127.0.0.1:9000;
        fastcgi_index index.php;
        fastcgi_keep_conn on;
        fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_send_timeout  3600;
        fastcgi_read_timeout 3600;
        fastcgi_connect_timeout 3600;
        send_timeout 3600;
        include fastcgi_params;
    }

}

23.7. proxy_pass directive

23.7.1. Basic

If proxy_pass is specified with a URI, then when a request is passed to the server, the part of a normalized request URI matching the location is replaced by a URI specified in the directive 

location /name/ {
    proxy_pass http://127.0.0.1/remote/;
# or http://127.0.0.1/
}

If proxy_pass is specified without a URI, the request URI is passed to the server in the same form as sent by a client when the original request is processed, or the full normalized request URI is passed when processing the changed URI 

location /some/path/ {
    proxy_pass http://127.0.0.1;
}

proxy_pass can have URI parts if it's under a non-regex location block 

location / {
  proxy_pass http://127.0.0.1:8888/long/uri;
# There's no trailing '/'
}

proxy_pass without URI under a regex location block will pass URI to the proxy 

location ~* \.(cfm|...|html)$) {
  proxy_pass http://127.0.0.1:8888;
}

proxy_pass with URI under a regex location block will throw an error. To avoid error, set a variable which holds the original or modified URI to pass 

location ~* \.(cfm|...|html)$) {
  set $request_url /site2$request_uri;
  if ($request_uri ~* ^/legacy_js) {
    set $request_url $request_uri;
  }
  proxy_pass http://127.0.0.1:8888$request_url;
}

23.7.2. Sample

location / {
  proxy_pass http://127.0.0.6:8888;
  proxy_set_header Host 127.0.0.6;
  proxy_redirect off;
  proxy_set_header X-Forwarded-Host $host;
  proxy_set_header X-Forwarded-Server $host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_connect_timeout 600;
  proxy_send_timeout 600;
  proxy_read_timeout 600;
  send_timeout 600;
}

Common docker container nginx:proxy:docker 

server {
  listen 80;
  server_name dev.myweb.com;

# refer to SSL config for nginx:ssl:server block for Let's Encrypt

  # deny .git
  location ~ /\.git {
    deny all;
  }

  location / {
    proxy_pass http://127.0.0.1:9977;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  # proxy_set_header X-Forwarded-SSL on; # for ssl
  # proxy_redirect default; # default is to redirect location into proxy_pass
  # proxy_redirect changes the response header ~Location~ from the proxied server
  }
}

$http_host equals always the HTTP_HOST request header. $host equals $http_host, lowercase and without the port number (if present), except when HTTP_HOST is absent or is an empty value.

  • In that case, $host equals the value of the server_name directive of the server which processed the request.

Refer to wordpress:ssl:behind proxy

proxy headers in PHP are HTTP_X_REAL_IP

23.7.3. Sample: Redirect a path to a different proxy nginx:redirect path to proxy

wp:redirect path to wp 

server {
  listen 80;
  server_name abc.com www.abc.com;

# this doens't work in wordpress! => location ^~ /subpath
  location ^~ /subpath/ {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Host $http_host;
    proxy_pass http://127.0.0.1:8787/;
  }

  location / {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Host $http_host;
    proxy_pass http://127.0.0.1:9977;
  }
}

23.8. Rewrite directive

  • http://nginx.org/en/docs/http/ngx_http_rewrite_module.html#rewrite
  • rewrite regex replacement [flag];
  • —
  • server, location, if
  • Escape
    • Don't need to escape forward slash /
    • See 8.6.9.2.1
  • https://regex101.com/r/xEWBb8/1/
    Regex type
    PCRE (PHP)
    Delimiter
    backtick `
    flags
    g i m (global, insensitive, multiline)

  • e.g. Source folder is CMS, and any case variants of CMS can be redirected 

    rewrite (?i)^/cms/?$ /CMS/index.cfm last;
rewrite (?i)^/cms/(.*)$ /CMS/$1 last;
# http://thissite.ca/ijk/xyz-123 => http://othersite.ca/citb/xyz
rewrite (?i)/([^/]+)-(\d+)/?$ https://othersite.ca/citb/$1;
  • Flags
    last
    Not to pass any more rewrite-module directives in the current server or location block
    redirect
    returns a temporary redirect with the 302 code; used if a replacement string does not start with http://, https://, or $scheme
    permanent
    same as redirect but 301

23.9. listen Directive

Default listen *:80; if Nginx is run as superuser or listen *:8000 otherwise.

Parameter ipv6only can be set only once across all config files. Default is on. Which means when [::] is used, listen only for IP v6. 

listen [::]:443 ssl ipv6only=on;

23.10. fastcgi directives - Module ngx_http_fastcgi_module

fastcgi_read_timeout
default 60s, 3600s

23.11. add_header

Context
http, server, location, if in location
Syntax
add_header name value [always]

Add to a response header. The value can contain variables.

always
By default the directive only adds header if the response code is in a certain range. always adds it regardless of the response code
add_header Content-Security-Policy "default-src * data: 'unsafe-eval' 'unsafe-inline'" always
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-XSS-Protection "1; mode=block" always;

23.12. Server name redirect

Use nginx:d:rewrite for regex redirect 

server {
  listen 80;
  listen [::]:80;
  server_name .mydomain.com;
# this matches *.mydomain.com
  return 301 http://www.adifferentdomain.com$request_uri;
}

# if you have ssl setup, you need to specify SSL info
# refer to nginx:ssl:example
server {
  # your original setting
  server_name .yoursite.com

  listen [::]:443 ssl ipv6only=on; # You might need to remove ipv6only=on; if it was set. managed by Certbot
  listen 443 ssl; # managed by Certbot
  ssl_certificate /etc/letsencrypt/live/yoursite.com/fullchain.pem; # managed by Certbot
  ssl_certificate_key /etc/letsencrypt/live/yoursite.com/privkey.pem; # managed by Certbot
  include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

  return 301 https://adifferentdoamin.com$request_uri;
}

23.13. error_log nginx:d:error_log

Default
error_log logs/error.log error;
Syntax
error_log file [level];
Context
main, http, mail, stream, server, location
Level
one of debug, info, notice, warn, error, crit, alert, or emerg

23.14. Core Module Directives

  • client_max_body_size
    • Maximum allowed size of the client request body. If the size in a request exceeds the configured value, the 413 (Request Entity Too Large) error is returned to the client. Please be aware that browsers cannot correctly display this error. Setting size to 0 disables checking of client request body size.
    • Default is 1m
    • Need to also change PHP post_max_size as well. Restart PHP and then Nginx
      • The following PHP configs may also need to be changed
        • memory_limit
        • upload_max_filesize
    • https://www.cyberciti.biz/faq/linux-unix-bsd-nginx-413-request-entity-too-large/

23.15. log_format and access_log nginx:d:log_format

log_format

Default
log_format combined "...too long";
Syntax
log_format name [escape=default|json|none] string ...;
Context
http
(no term)
Format variables
$msec
time in seconds with a milliseconds resolution at the time of the log write
$request_time
request processing time in seconds with a milliseconds resolution; time elapsed between the first bytes were read from the client and the log write after the last bytes were sent to the client
$upstream_connect_time
time spent establishing a connection with an upstream server
$upstream_header_time
time between establishing a connection to an upstream server and receiving the first byte of the response header
$upstream_response_time
tiem between establishing a connection to an upstream server and receiving the last byte of the response body
$status
response status

Format combined has this 

log_format combined '$remote_addr - $remote_user [$time_local] '
            '"$request" $status $body_bytes_sent '
            '"$http_referer" "$http_user_agent"';

access_log

Default
access_log logs/access.log combined;
Syntax
access_log path [format [buffer=size] [gzip[=level]] [flush=time] [if=condition]];
Or
access_log off;
format is not specified
then combined is used
Context
http, server, location, if in location, limit_except

23.15.1. Use Linux commands to analyze

cd /var/log/nginx

# Filter archived logs with string `entity=loans&action=list`, grab the 17th column, sort and count the unique values
zcat access.log.*.gz | grep 'entity=loans&action=list' | awk '{print $17}' | sort | uniq -c | sort -rn

# cut -d '"' -f3 :: delimit the line with double quote and choose the 3rd element
cat access.log | cut -d '"' -f3 | cut -d ' ' -f2 | sort | uniq -c | sort -rn

23.16. Let's Encrypt SSL nginx:ssl letsencrypt:certbot

  • On Ubuntu, install Let's Encrypt SSL client certbot package to obtain, install and renew certificate
  • means to modify webserver config e.g. Nginx /etc/nginx/sites-available/mywebsite.conf
  • means to get and store certificates to /etc/letsencrypt directory
    all live certificates
    /etc/letsencrypt/live/example.com/fullchain.pem
  • used to be letsencrypt in older version or certbot-auto in an alternate installation method
    • Syntax certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...

    • Commands

      run
      obtain and install
      certonly
      obtain or renew a certificate
      (no term)
      renew
      Dry run to renew all certificates
      certbot renew --dry-run
      • certificates
      Check all certificates status
      certbot certificates

    • Options

      --webroot -w MYWEBROOT_PATH
      letsencrypt:certbot:webroot
      -n
      non-interactive
      --agree-tos
      agree to ACME server's Subscriber Agreement
      --force-renewal, --renew-by-default
      If a cert exists, renew it regardless of whether it's near expiry
      --dry-run
      dry run
      (no term)
      A plugin can be an authenticator and/or an installer. Use options to specify which plugin to use
      • -a, --authenticator

      • -i, --installer Different plugins can be combined 

        certbot run -a webroot -i apache -w /var/www/html -d example.com
  • /etc/letsencrypt/renewal/mydomain.com.conf
  • /etc/letsencrypt/cli.ini (usually empty)
  • /var/log/letsencrypt
  • it runs certbot -q renew so renewal config file is very important for each domain

23.16.1. Challenges

http challenge (port 80)
requires to place a file with specific name and content in your-website-root/.well-known/acme-challenge/ directoy
dns (port 53 open on DNS server)
requires to place a TXT DNS record with specific contents
  • e.g. _acme-challenge.example.com. 300 IN TXT "gfj9Xq…Rg85nM"
tls-sni (port 443)
prepares a self-signed SSL certificate with the challenge validation encoded into a subjectAlternatNames entry. Config your SSL server to present this challenge SSL certificate to the ACME server using SNI.
  • nginx and apache plugins use this type

23.16.2. certbot nginx apache plugins

They are authenticator and install plugins. tls-sni-01 challenge 

sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install python-certbot-nginx
# sudo apt-get install python-certbot-apache
  • Allow HTTPS in firewall nginx:ufw

  • Obtain an SSL certificate using --nginx plugin for those -d server names 

    sudo certbot --nginx -d example.com -d www.example.com
  • certbot looks for server block files with server_name
  • The first time running certbot, prompt to enter an email address and agree to the terms of service. After doing so, certbot will communicate with the Let's Encrypt server, then run a challenge to verify that you control the domain you're requesting a certificate for
  • After that, certificate is downloaded, Nginx config is updated and reloaded

  • /etc/nginx/sites-enabled/yoursite.com nginx:ssl:server block 

    server {
listen 80;
listen [::]:80;
server_name example.com www.example.com; 
return 301 https://example.com$request_uri;
  # force redirect to https
}

server {
  # your original setting
  # listen 80; # if you want to serve http as well in the same block
  # ssl on; # is not needed when ssl is in listen directive
  listen 443 ssl; # managed by Certbot
  ssl_certificate /etc/letsencrypt/live/yoursite.com/fullchain.pem; # managed by Certbot
  ssl_certificate_key /etc/letsencrypt/live/yoursite.com/privkey.pem; # managed by Certbot
  include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

  # follow by these proxy rules nginx:proxy:docker

}

# certbot will insert this block, but it's better to use the one atop.
server {
  if ($host = yoursite.com) {
    return 301 https://$host$request_uri;
} # managed by Certbot

  listen 80;
  server_name yoursite.com;
return 404; # managed by Certbot
}
  • /etc/letsencrypt/live/example.com/fullchain.pem
  • /etc/letsencrypt
  • ssl:test
  • Let's Encrypt SSL certificates are valid for 90 days. On Ubuntu, install certbot package to renew certificate. certbot renew is run twice a day on systemd timer. For non-systemd, /etc/cron.d also runs twice a day.

  • /etc/cron.d/certbot 

    SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(3600))' && certbot -q renew
  • Dry run sudo certbot renew --dry-run, expect no error

23.16.3. manual plugin

authenticator only
obtain a certificate by giving you instructions to perform domain validation yourself. Additionally allows you to specify scripts to automate the validation task in a customized way. hooks can be specified
Challenge types
http-01, dns-01 or tls-sni-01

Example usage for HTTP-01: 

certbot certonly --manual --preferred-challenges=http --manual-auth-hook /path/to/http/authenticator.sh --manual-cleanup-hook /path/to/http/cleanup.sh -d secure.example.com

authenticator.sh 

#!/bin/bash
dir=/path/to/web-root/.well-known/acme-challenge
mkdir -p $dir
filename=$dir/$CERTBOT_TOKEN
test -f $filename || touch $filename
echo $CERTBOT_VALIDATION > $filename
chown -R www-data:www-data $filename

cleanup.sh 

rm -f /var/www/htdocs/.well-known/acme-challenge/$CERTBOT_TOKEN

Environment vars available to hooks

CERTBOT_DOMAIN
The domain being authenticated
CERTBOT_VALIDATION
The validation string (HTTP-01 and DNS-01 only)
CERTBOT_TOKEN
Resource name part of the HTTP-01 challenge (HTTP-01 only)
CERTBOT_CERT_PATH
The challenge SSL certificate (TLS-SNI-01 only)
CERTBOT_KEY_PATH
The private key associated with the aforementioned SSL certificate (TLS-SNI-01 only)
CERTBOT_SNI_DOMAIN
The SNI name for which the ACME server expects to be presented the self-signed certificate located at $CERTBOT_CERT_PATH (TLS-SNI-01 only)
CERTBOT_AUTH_OUTPUT
(cleanup hook only) Whatever the auth script wrote to stdout
23.16.3.1. Example usage for DNS-01

(Cloudflare API v4) (for example purposes only, do not use as-is) 

certbot certonly --manual --preferred-challenges=dns --manual-auth-hook /path/to/dns/authenticator.sh --manual-cleanup-hook /path/to/dns/cleanup.sh -d secure.example.com

authenticator.sh 

#!/bin/bash

# Get your API key from https://www.cloudflare.com/a/account/my-account
API_KEY="your-api-key"
EMAIL="your.email@example.com"

# Strip only the top domain to get the zone id
DOMAIN=$(expr match "$CERTBOT_DOMAIN" '.*\.\(.*\..*\)')

# Get the Cloudflare zone id
ZONE_EXTRA_PARAMS="status=active&page=1&per_page=20&order=status&direction=desc&match=all"
ZONE_ID=$(curl -s -X GET "https://api.cloudflare.com/client/v4/zones?name=$DOMAIN&$ZONE_EXTRA_PARAMS" \
     -H     "X-Auth-Email: $EMAIL" \
     -H     "X-Auth-Key: $API_KEY" \
     -H     "Content-Type: application/json" | python -c "import sys,json;print(json.load(sys.stdin)['result'][0]['id'])")

# Create TXT record
CREATE_DOMAIN="_acme-challenge.$CERTBOT_DOMAIN"
RECORD_ID=$(curl -s -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
     -H     "X-Auth-Email: $EMAIL" \
     -H     "X-Auth-Key: $API_KEY" \
     -H     "Content-Type: application/json" \
     --data '{"type":"TXT","name":"'"$CREATE_DOMAIN"'","content":"'"$CERTBOT_VALIDATION"'","ttl":120}' \
         | python -c "import sys,json;print(json.load(sys.stdin)['result']['id'])")
# Save info for cleanup
if [ ! -d /tmp/CERTBOT_$CERTBOT_DOMAIN ];then
    mkdir -m 0700 /tmp/CERTBOT_$CERTBOT_DOMAIN
fi
echo $ZONE_ID > /tmp/CERTBOT_$CERTBOT_DOMAIN/ZONE_ID
echo $RECORD_ID > /tmp/CERTBOT_$CERTBOT_DOMAIN/RECORD_ID

# Sleep to make sure the change has time to propagate over to DNS
sleep 25

cleanup.sh 

#!/bin/bash

# Get your API key from https://www.cloudflare.com/a/account/my-account
API_KEY="your-api-key"
EMAIL="your.email@example.com"

if [ -f /tmp/CERTBOT_$CERTBOT_DOMAIN/ZONE_ID ]; then
    ZONE_ID=$(cat /tmp/CERTBOT_$CERTBOT_DOMAIN/ZONE_ID)
    rm -f /tmp/CERTBOT_$CERTBOT_DOMAIN/ZONE_ID
fi

if [ -f /tmp/CERTBOT_$CERTBOT_DOMAIN/RECORD_ID ]; then
    RECORD_ID=$(cat /tmp/CERTBOT_$CERTBOT_DOMAIN/RECORD_ID)
    rm -f /tmp/CERTBOT_$CERTBOT_DOMAIN/RECORD_ID
fi

# Remove the challenge TXT record from the zone
if [ -n "${ZONE_ID}" ]; then
    if [ -n "${RECORD_ID}" ]; then
    curl -s -X DELETE "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$RECORD_ID" \
        -H "X-Auth-Email: $EMAIL" \
        -H "X-Auth-Key: $API_KEY" \
        -H "Content-Type: application/json"
    fi
fi

23.16.4. certbot standalone plugin

An authenticator plugin uses a standalone webserver to obtain a certificate. It's useful when there's no webserver. Need to open port 80 or 443 http-01 or tls-sni-01 challenge.

https://www.digitalocean.com/community/tutorials/how-to-use-certbot-standalone-mode-to-retrieve-let-s-encrypt-ssl-certificates

sudo add-apt-repository ppa:certbot/certbot sudo apt-get update sudo apt-get install certbot

Certbot may run as a standalone webserver, allow either 80 or 443 sudo ufw allow 80

generete certificate only sudo certbot certonly –standalone –preferred-challenges http -d example.com

The certbot package we installed takes care of this for us by adding a renew script to /etc/cron.d

/etc/letsencrypt/renewal/example.com.conf renew_hook = systemctl reload rabbitmq

sudo certbot renew –dry-run

23.16.5. webroot plugin --webroot -w letsencrypt:certbot:webroot

authenticator only
obtain certificate and write to an existing webroot without webserver config mod
Challenge types
http-01
-w
alias --webroot-path MYWEBROOT_PATH
certbot certonly --webroot -w /var/www/example/ -d www.example.com -d example.com -w /var/www/other -d other.example.net -d another.other.example.net

23.16.6. manage and renew certificates

/etc/letsencrypt/archive and /etc/letsencrypt/keys contain all previous keys and certificates, while /etc/letsencrypt/live symlinks to the latest versions. 

# check status for all certificates
certbot certificates
certbot certonly --cert-name example.com

# Update an existing certificate with a new certificate that contains all old domains and new domains
# Use -d to specify all existing and new domains.
certbot --expand -d existing.com,example.com,newdomain.com

# remove some domains from an exisiting certificate. before it contains example.com and www.example.com
certbot certonly --cert-name example.com -d example.com

# overwrites all domains for an existing certificate
certbot certonly --cert-name example.com -d example.org,www.example.org

# revoke
certbot revoke --cert-path /etc/letsencrypt/live/CERTNAME/cert.pem

# renew any existing certificates that expire in less than 30 days.
# --pre-hook and --post-hook hooks run before and after every renewal attempt.
certbot renew --pre-hook "service nginx stop" --post-hook "service nginx start"

# If you want your hook to run only after a successful renewal, use --deploy-hook in a command like this.
certbot renew --deploy-hook /path/to/deploy-hook-script
23.16.6.1. Hooks
certbot renew --pre-hook "service nginx stop" --post-hook "service nginx start"

# If you want your hook to run only after a successful renewal, use --deploy-hook in a command like this.
certbot renew --deploy-hook /path/to/deploy-hook-script

deploy-hook-script 

#!/bin/sh

set -e

for domain in $RENEWED_DOMAINS; do
        case $domain in
        example.com)
                daemon_cert_root=/etc/some-daemon/certs

                # Make sure the certificate and private key files are
                # never world readable, even just for an instant while
                # we're copying them into daemon_cert_root.
                umask 077

                cp "$RENEWED_LINEAGE/fullchain.pem" "$daemon_cert_root/$domain.cert"
                cp "$RENEWED_LINEAGE/privkey.pem" "$daemon_cert_root/$domain.key"

                # Apply the proper file ownership and permissions for
                # the daemon to read its certificate and key.
                chown some-daemon "$daemon_cert_root/$domain.cert" \
                        "$daemon_cert_root/$domain.key"
                chmod 400 "$daemon_cert_root/$domain.cert" \
                        "$daemon_cert_root/$domain.key"

                service some-daemon restart >/dev/null
                ;;
        esac
done

Script files can be placed at /etc/letsencrypt/renewal-hooks/pre, /etc/letsencrypt/renewal-hooks/deploy, and /etc/letsencrypt/renewal-hooks/post Script files are run in alpha order.

23.16.6.2. Renewal conf

Sample /etc/letsencrypt/renewal/mydomain.com.conf 

# renew_before_expiry = 30 days
version = 0.21.1
archive_dir = /etc/letsencrypt/archive/mydomain.com
cert = /etc/letsencrypt/live/mydomain.com/cert.pem
privkey = /etc/letsencrypt/live/mydomain.com/privkey.pem
chain = /etc/letsencrypt/live/mydomain.com/chain.pem
fullchain = /etc/letsencrypt/live/mydomain.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = abcfdfsda
installer = nginx
authenticator = nginx

23.16.7. A certificate nginx:ssl:example

server {
  # your original setting
  server_name .yoursite.com

  listen [::]:443 ssl ipv6only=on; # You might need to remove ipv6only=on; if it was set. managed by Certbot
  listen 443 ssl; # managed by Certbot
  ssl_certificate /etc/letsencrypt/live/yoursite.com/fullchain.pem; # managed by Certbot
  ssl_certificate_key /etc/letsencrypt/live/yoursite.com/privkey.pem; # managed by Certbot
  include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

  # set proxy
}
certbot renewal config file
/etc/letsencrypt/renewal/mydomain.com.conf
global config
/etc/letsencrypt/cli.ini

23.16.8. Windows IIS

  • IIS 8 on Windows Server 2012 or IIS 8.5 on Windows Server 2012 R2.
  • IIS 10 on Windows server 2016

Manual Install SSL on IIS 8 or IIS 10 :: create CSR on IIS, submit and get SSL, import cert.

  • Or on IIS, import the .p12 file with key and crt from Linux

IIS Binding

Server Binding
IP:Port:HostHeader to a website. e.g. *:80:* *:81:* 192.168.1.1:80:* *:80:www.microsoft.com *:80:microsoft.com
(no term)
HTTPS is a server binding with SSL certificate *:443:abc.com *:443:www.abc.com. For IIS 2012+, enable Server Name Indication option to bind multiple SSL certificates to a single IP address. Before, IIS can only bind a single SSL cert. to one IP address

IIS Websites

  • Path (a container of physical and virtual directories) + Server Binding = Website. e.g. default container is %systemdrive%\inetpub\wwwroot

win-acme (ACME client) Intro GitHub

Download latest release, unzip and run letsencrypt.exe with admin privileges.

  • A task is added to Windows Task Scheduler

23.16.9. Alternative ACME clients

  • https://www.digitalocean.com/community/tutorials/an-introduction-to-let-s-encrypt
  • Automatic Certificate Management Environment. It is a protocol. Let's Encrypt just issues certificates
  • Cerbot is one ACME client that is developed by Electronic Frontier Foundation (EFF) which uses ACEM protocol
  • lego">Written in Go, lego is a one-file binary install, and supports many DNS providers when using the DNS challenge
  • acme.sh">acme.sh is a simple shell script that can run in unprivileged mode, and also interact with 30+ DNS providers
  • Caddy">Caddy is a full web server written in Go with built-in support for Let's Encrypt.

All other clients

23.16.10. Limitation

  • 5 levels of validation
    Domain Validation DV
    domain registrar or put a file on the domain
    (no term)
    Organization Validation OV
    Extended Validation EV
    phone call. Most expensive
    (no term)
    Wildcard Certificate
  • Certificate Authority (CA) provides the above validation and Commercial Certificate Authorities are the only one provide EV and Wildcard
  • Let's Encrypt provides DV only and planning to support Wildcard in near future. A certificate with up to 100 hostnames

23.16.11. SSL config nginx:ssl:config

23.16.11.1. Combine end-user and bundled intermediate certificates
cat your_domain.crt intermediate.crt root.crt >> ssl-bundle.crt

# e.g. PositiveSSL
cat example_com.crt COMODORSADomainValidationSecureServerCA.crt COMODORSAAddTrustCA.crt AddTrustExternalCARoot.crt >> ssl-bundle.crt

# e.g. combine end-user and bundled intermediate certificates
cat example_com.crt bundle.crt >> ssl-bundle.crt
23.16.11.2. Strengthen Nginx SSL Setup
# Protocol support
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

# Perfect when only 1.2 is used
ssl_protocols TLSv1.2;

# Put it in server block
ssl_dhparam ssl/dhparam.pem;

# Generate DHE
openssl dhparam -out /etc/nginx/ssl/dhparam.pem 4096

# Specify ciphers that are enabled on Nginx
# Default ssl_ciphers HIGH:!aNULL:!MD5;

# Change it to use openssl ciphers, run this to get ciphers provided by openssl and wrap it with double quotes and assign it to ssl_ciphers
# openssl ciphers

ssl_ciphers "AES256+EECDH:AES256+EDH:!aNULL;"
ssl_prefer_server_ciphers on;

# Default: ssl_session_cache none;
# shared :: a cache shared between all Nginx worker processes. In bytes. 1 mb can store about 4000 sessions. le_nginx_SSL is a cache name and can be used in multiple virtual servers
# Default: ssl_session_timeout 5m;

ssl_session_cache shared:le_nginx_SSL:1m;
ssl_session_timeout 1440m;

# SSL Stapling (not implemented by Let's Encrypt)
ssl_stapling on;
ssl_stapling_verify on;

23.17. nginxconfig.io

server {
    listen 80;
    listen [::]:80;

    server_name .myweb.ca;

    include nginxconfig.io/letsencrypt.conf;

    # letsencrypt.conf
    # location ^~ /.well-known/acme-challenge/ {
    #    root /var/www/_letsencrypt;
    # }

    location / {
    return 301 https://www.myweb.ca$request_uri;
    }

}

nano /etc/nginx/nginx.conf
nano  /etc/nginx/sites-available/myweb.ca.conf
nano /etc/nginx/nginxconfig.io/php_fastcgi.conf
nano  /etc/nginx/nginxconfig.io/wordpress.conf
nano  /etc/nginx/nginxconfig.io/general.conf
ln -s /etc/nginx/sites-available/myweb.ca.conf /etc/nginx/sites-enabled/myweb.ca.conf

openssl dhparam -dsaparam -out /etc/nginx/dhparam.pem 2048
# create dir
sudo -u www-data sh -c "mkdir -p /var/www/_letsencrypt"~

chown www-data:www-data /var/www/_letsencrypt

# comment out all ssl_* directives - add `#;` in front
sed -i -r 's/(listen .*443)/\1;#/g; s/(ssl_(certificate|certificate_key|trusted_certificate) )/#;#\1/g' /etc/nginx/sites-available/myweb.ca.conf

# create cert only
certbot certonly --webroot -d myweb.ca -d www.myweb.ca --email ssl@myweb.ca -w /var/www/_letsencrypt -n --agree-tos --force-renewal --dry-run

# add back ssl_* directives - remove `#;` in front
sed -i -r 's/#?;#//g' /etc/nginx/sites-available/myweb.ca.conf

23.18. deny nginx:d:deny

Syntax: deny address | CIDR | unix: | all; Default: — Context: http, server, location, limit_except 

location ~* \.(git|rb|inc|ht)$ {
    deny all;
}

23.19. Redirect a website to another website

23.19.1. HTTPS to HTTPS redirect

  • Redirect https://mywebsite.com to https://mywebsite2.ca. Both websites are not hosted on this Nginx server

  • /etc/nginx/sites-available/mywebsite.com.conf 

    server {
listen 443 ssl http2;
listen [::]:443 ssl http2;

server_name mywebsite.com www.mywebsite.com;
set $base /var/www/mywebsite.com;
root $base/public;

# SSL
ssl_certificate /etc/letsencrypt/live/mywebsite.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mywebsite.com/privkey.pem;
ssl_trusted_certificate /etc/letsencrypt/live/mywebsite.com/fullchain.pem;

# index.php fallback
# remove comment after certificate is installed

#location / {
#   return 302 https://www.mywebsite2.ca;
#}

}

# non-www, subdomains redirect
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;

server_name .mywebsite.com;

# SSL
ssl_certificate /etc/letsencrypt/live/mywebsite.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mywebsite.com/privkey.pem;
ssl_trusted_certificate /etc/letsencrypt/live/mywebsite.com/fullchain.pem;

return 302 https://www.mywebsite2.ca;
}

# HTTP redirect
server {
listen 80;
listen [::]:80;

server_name .mywebsite.com;

include nginxconfig.io/letsencrypt.conf;

location / {
    return 302 https://www.mywebsite2.ca;
}
}
  • Create an index.html /var/www/mywebsite.com/public/index.html
  • Change DNS A @ and CNAME www or A www record for mywebsite.com to point to Nginx
    • Go to http://mywebsite.com and http://www.mywebsite.com, Nginx Welcome page should show
    • DNS is propagated!

  • Run 

    ln -s /etc/nginx/sites-available/mywebsite.com.conf /etc/nginx/sites-enabled/mywebsite.com.conf

sed -i -r 's/(listen .*443)/\1;#/g; s/(ssl_(certificate|certificate_key|trusted_certificate) )/#;#\1/g' /etc/nginx/sites-available/mywebsite.com.conf

systemctl reload nginx

# dry run to review errors
certbot certonly --webroot -d mywebsite.com -d www.mywebsite.com --email ssl@gmail.com -w /var/www/_letsencrypt -n --agree-tos --force-renewal --dry-run

sed -i -r 's/#?;#//g' /etc/nginx/sites-available/mywebsite.com.conf

systemctl reload nginx

23.19.2. HTTP to HTTPS redirect nginx:redirect:HttpToHttps

server {
# we want to listen on port 80 on all IPs on our system - both IPv4 and IPv6
    listen 80;
    listen [::]:80;
# our primary server name is the first, aliases simply come after it. you can also include wildcards like *.example.com
    server_name source.com www.source.com source.ca www.source.ca;

    location = /special-page {
    return 302 http://www.target.ca;
    }

    location ~ ^/(news|magazine-archives) {
    rewrite (?i)\/([^\/]+)-(\d+)\/?$ https://www.target.ca/source/$1 last;
    }
    location / {
    return 302 https://www.target.com/citb;
    }

    # define our access and error logs for this vhost
    # access_log /var/log/nginx/access-sourceweb.log;
    # error_log /var/log/nginx/error-sourceweb.log;
}

24. Git

24.1. Show config

# 3 levels: system < global (user, ~/.gitconfig) < local (./.git/config) < ./.git/info/attributes < ./.gitattributes

# List all active config for the current folder
git config --list | grep credential
git config --get core.autocrlf

# List global config only
git config --global --list

# Edit or check all global config
git config --global --edit

# Get all files that forms each config, also good to locate config files
git config --list --show-origin

# Edit system config file requires admin permission. Use notepad to change 
# C:\\ProgramData/Git/config
# core.symlinks = false

# edit system config
# Windows :: C:/Program Files/Git/mingw64/etc/gitconfig
# Mac Homebrew ::
#   - Source :: /opt/homebrew/etc/gitconfig
#   - Old :: /usr/local/etc/gitconfig
git config --edit --system

# edit global config which is ~/.gitconfig
# Mac /Users/lili/.gitignore
git config --edit --global

# Find path to git.exe
which git.exe

24.1.1. User name and email

# change username and email in global
git config --global user.name "Li Li"
git config --global user.email "a@b.ca"

# change username and email for the current repo
git config user.email myname@abc.omc
git config user.name "My Name"

# set temporary config for commit
git -c user.name='Li Li' -c user.email='a@a.ca' commit -m '...'

# or
git commit -m '...' --author="Li Li <a@b.ca>"

Don't verify SSL and use temporary username and password to pull and push 

git -c http.sslVerify=false -c user.name=abc -c user.password=xyz pull -p

24.1.2. .gitattributes

  • ./.git/info/attributes or ./.gitatrributes
# All files are checked into the repo with LF
# text :: It marks the path as a text file, which enables end-of-line conversion: When a matching file is added to the index, the file’s line endings are normalized to LF in the index. Conversely, when the file is copied from the index to the working directory, its line endings may be converted from LF to CRLF depending on the eol attribute, the Git config, and the platform
# -text :: Unsetting the text attribute on a path tells Git not to attempt any end-of-line conversion upon checkin or checkout. Equivalent to core.autocrlf = false
# text=auto :: Git decides by itself whether the file is text or binary. If it is text and the file was not already in Git with CRLF endings, line endings are converted on checkin and checkout as described above. Otherwise, no conversion is done on checkin or checkout.
* -text

# These files are checked out using CRLF locally
*.bat eol=crlf

# List files in the index and the working tree
# --eol will show i/<eolinfo><SPACES>w/<eolinfo><SPACES>attr/<eolattr><SPACE*><TAB><file>
#   - <eolinfo> is the file content identification used by Git when the "text" attribute is "auto" (or not set and core.autocrlf is not false). <eolinfo> is either "-text", "none", "lf", "crlf", "mixed" or "". "" means the file is not a regular file, it is not in the index or not accessible in the working tree.
#   - <eolattr> is the attribute that is used when checking out or committing, it is either "", "-text", "text", "text=auto", "text eol=lf", "text eol=crlf". Since Git 2.10 "text=auto eol=lf" and "text=auto eol=crlf" are supported
#   - Both the <eolinfo> in the index ("i/<eolinfo>") and in the working tree ("w/<eolinfo>") are shown for regular files, followed by the ("attr/<eolattr>").
git ls-files --eol

24.2. Alias 

# git co yourbranch
git config --global alias.co checkout
git config --global alias.br branch
git config --global alias.st status
git config --global alias.sti 'status --ignored'
git config --global alias.acp '!llg_acp(){ git add . && git commit -m "$1" && git push; };llg_acp'

~/.gitconfig 

[alias]
     co = checkout
     acp = "!llg_acp(){ git add . && git commit -m \"$1\" && git push; };llg_acp"
     commit-x = -c user.name='Change to X Author' -c user.email='x@a.ca' commit

# Or include a separate file for alias
[include]
    path = /Users/lili/.gitconfig.alias.conf

# .gitconfig.alias.conf
[alias]
    # ...

24.3. Working Tree

  • Working Copy (unstaged, working directory, working tree) -> git add -> Index (staged) -> git commit -> HEAD
  • ">cat .git/HEAD
    • It can be the tip of a branch
    • Or any commit in any branch
      • Use git checkout <branchname> to set HEAD to the tip of a branch
  • the previous commit
  • it was never staged nor committed
  • it was previously committed, but it is not staged
  • ready for commit
  • file is changed but not staged

24.4. Archive

  • Zip to a file 

    git archive [--format=<fmt>] [--list] [--prefix=<prefix>/] [<extra>]
	  [-o <file> | --output=<file>] [--worktree-attributes]
	  [--remote=<repo> [--exec=<git-upload-archive>]] <tree-ish>
	  [<path>…​]
    • git archive --format=zip -o /full/path/to/zipfile.zip master
  • d:tar or zip
  • prepend <prefix>/ to each filename in the archive
  • -o <file>, --output=<file>

  • Export files from master branch to a folder without compression 

    git archive master | tar -x -C /export/to/thisfolderwithoutcompression

24.5. diff

  • git:stages
  • Between Working Copy and Index (Staged)
  • Between HEAD and Index (Staged)
  • Between (Unstaged or Staged) and HEAD
  • Between (Unstaged or Staged) and HEAD^
  • git:branch:file difference

24.6. Patch

  • Create a patch
    • Refer to git:diff
    • git diff > my.patch
    • git diff --cached > my.patch
    • git format-patch -10
    • git fortmat-patch -10 --stdout > my.patch
  • Apply a patch
    Apply a patch to working directory
    git apply my.patch

24.7. Show file names only between 2 commits

Show file names only for 1 commit 

git diff-tree --no-commit-id --name-only -r "<SHA>"

Show file names only between 2 commits 

git --no-pager diff --name-only SHA1 SHA2

24.8. Stash, Reset, Clean

  • git:stages
  • git reset
    • eq. to git reset --mixed HEAD
      • Reset current HEAD to a <tree-ish> or <commit> which defaults to HEAD
      • default mode. Reset the Index (staged) but not the working tree
  • See 1.6
  • ">remove untracked files
  • Save and remove Staged and Unstaged changes
    Stash tracked file changes
    git stash
    Give a name
    git stash save 'my stash'
    Stash tracked file changes and untrack files
    git stash --include-untracked
    List all stashes
    git --no-pager stash list
    Apply latest stash
    git stash apply
    Apply 2nd latest stash
    git stash apply "stash@{2}"
    (no term)
    Apply when there's no staged and unstaged
    (no term)
    After applied, all changes are unstaged
    Remove latest stash
    git stash drop "stash@{0}"
    remove all stashes
    git stash clear

24.9. Checkout vs Reset vs Revert

  • Current branch release has c1, c2 and c3 commits in order
    git checkout c1
    sets branch to commit 1 in a detached HEAD state. HEAD is not at tip of branch. It's at that commit. Not changing anything
    • You can go back to the branch HEAD by git checkout release. Then it's no longer in detached HEAD state.
    git checkout <branchname>
    go back to the tip of the branch
    git checkout c1 . or git checkout -f c1 -- .
    The working directory is set to the same as c1. Changes to be committed are the changes needed to go back from c3 to c1. The HEAD is still at c3. If you commit as c4 (which is after c3), c4 will be the same as c1.
    • To undo this, git reset --hard
    git checkout -- path/to/file
    discard changes for a file and revert the file to the last committed
    Checkout a file from another branch
    git checkout mybranchname -- path/to/file
    git reset c1
    sets branch HEAD to commit 1
    • Commits beyond c1 to the previous HEAD are saved in reflog
    (no term)
    git reset --hard
    • eq. to git reset --hard HEAD
    • clean working tree, go back to HEAD but the previously untracked changes are intact
    git reset --hard HEAD^
    set branch HEAD to the second last commit
    git reset HEAD^
    set branch HEAD to the second last commit, the last commit changes are in working tree
    (no term)
    git reset rewrites history. Only use it for local changes that are never pushed to remote
    git revert c2 --no-edit
    undo commit c1 and creates a commit. --no-edit means to skip the commit message
    • https://mirrors.edge.kernel.org/pub/software/scm/git/docs/howto/revert-a-faulty-merge.txt
    • If c2 is a merge commit, it will have more than 1 parents

      • Running git log c2 shows all sub commits that are merged in c2 merge commit. git log c2 -1 only shows the merge commit. Something like this shows 

        commit c2
Merge: parent1 parent2
Author: Li Li <li@li.com>
Date:   Mon Jun 24 11:36:52 2024 -0400

    Merge branch 'feature-branch' into release

        In this case, parent1 is original commit in the feature-branch, parent2 is c1 in the current release branch, but it is better for you always check. To revert c2 on release branch, git revert c2 -m 1 where -m 1 is to choose parent1

24.10. Checkout a folder from another branch to current branch

current branch is master and get folder target-folder/ from feature branch 

git checkout master
git checkout feature target-folder/

24.11. Commit

Amend last commit message
git commit --amend -m "new commit message"
Amend last commit's author
git commit --amend --author="FirstName LastName <a@b.ca>"
  • After changing author, it still keeps who initially makes the commit
(no term)

If the commit is not the last commit on HEAD.. Do this but it will change history from that commit onwards 

# bbc643cd is the commit that you want to change, ^ means rebase back to the commit before the one you wish to modify
git rebase --interactive 'bbc643cd^'

# for commit bbc643cd, change `pick` to `edit`
# you can pick more commits to edit/amend

# then modify author
git commit --amend --author="MA Authors <li@mortgageautomator.com>" --no-edit

git rebase --continue

# rebase will stop at the next commit that you previously mark as edit
# rebase continue until it ends

git push -f

24.12. Branch

24.12.1. Current branch info

# all local branches and their last commit message, current branch is * highlighted
git branch -v

# all local branches and their last commit message and the commit's remote branch (upstream)
git branch -vv

# current branch and its upstreams
# -s :: short
# -b :: branch and its tracking info (upstream)
# This shows the current local branch has a corresponding upstream branch
# ## release..origin/release
# This shows the current local branch does not have a corresponding upstream branch
# ## release
git status -sb

24.12.2. Duplicate a remote branch as a new local branch

git checkout -b my-new-local-branch origin/my-remote-branch

24.12.3. Duplicate current branch with uncommited changes and switch to it

git checkout -b newbranch

Create a local branch and then push to a new remote branch. Save changes to a new branch. 

git checkout -b newbranch
git add . && git commit -m "..."

git push --set-upstream origin newbranch
# eq.
git push -u origin newbranch
# eq. set current local branch's upstream without specifying the local branch name
git push -u origin HEAD

24.12.4. Duplicate current branch to a commit and switch

git checkout -b newbranch <SHA1>

24.12.5. Delete branch

git checkout master
you can't be on the branch that is going to be deleted
(no term)
git branch -d branchgroup/branchname
  • delete the local branch only if it has been pushed and merged with the remote branch
(no term)
git branch -D branchname
  • delete the local branch even if it hasn't been pushed or merged with the remote branch
(no term)
Undo branch delete
git branch branchName <sha1>
the sha1 shows after the branch is deleted
(no term)
git push origin --delete branchgroup/branchname
  • delete a branch remotely
  • ~git push origin :branchgroup/branchname
    • error: unable to push to unqualified destination: remoteBranchName The destination refspec neither matches an existing ref on the remote nor begins with refs/, and we are unable to guess a prefix based on the source ref. error: failed to push some refs to 'git@repository_name'
    • This error shows if the remote branch is already deleted
git fetch -p
--prune, -p purge/prune. Before fetching, remove any remote-tracking references that no longer exist on the remote
  • On local there's origin/abc but remote does not have branch abc, then local origin/abc will be removed

24.12.6. Rename branch

  • Branch name can contain -
    Restrain from using
    , .
  • Branch name better not have
    • :
# Rename current branch
git branch -m new-branch-name

# Rename another branch which is not the current branch
git branch -m old-branch-name new-branch-name

# If the previous local branch has pushed to remote, do the above to rename the local branch, and do these
# Delete the old remote branch and push the current branch to the new remote branch
git push origin :old-branch-name new-branch-name
# Reset the upstream of the local branch to the new remote branch
git push origin -u new-branch-name

24.12.7. Find branches which contain a commit

git branch --contains mycommithash

24.12.8. Commit difference between 2 branches

What commits li has but master doesn't
git log master..li
  • For a specific file git log master..li path/to/file
    • Current branch
  • git log master..HEAD
    • git log master..@

  • For multiple branches, e.g. to see what commits in (feature-1-branch and feature-2-branch) but are not on (release nor master)

    • git log --no-merges feature-1-branch feature-2-branch ^release ^master
    • To limit to a file

    git log --no-merges feature-1-branch feature-2-branch ^release ^master -- path/to/file

(no term)
What commits a) li has but master doesn't and b) master has but li doesn't
  • git log master...li
(no term)
Options
-p
show changed files for each commit listed
  • git log -p master..li
(no term)
See git:branch:file difference
(no term)
See 24.14

24.12.9. File difference between 2 branches

  • git:diff
  • git diff --name-status master..origin/master
    D or -
    Missing file or line in remote branch
    A or +
    New file or line in remote branch
    M
    Both are modified
  • git diff --stat --color master..origin/master
    • Only number of lines of difference, can't see which files are D (deleted), A (added) or M (modified)
  • See git:log:commit difference

24.12.10. Find authors of all remote/local branches and tags

git for-each-ref --format='%(committerdate) %09 %(authorname) %09 %(refname)' | sort -k5n -k2M -k3n -k4n

24.12.11. Make current branch exactly as another branch

Make current branch li exactly the same as remote branch origin/master. 

git checkout li
git reset --hard origin/master

Pull forced after push forced 

git checkout li
git rebase main
git push -f

# On another computer
git checkout li
git fetch
git reset --hard origin/li

24.12.12. Force push to remote branch

local master has different commits than remote li's master branch 

git checkout master
git push li master -f

24.12.13. Find most recent ancestor of 2 branches. See 24.14

24.12.14. Revert to common ancestor

You have a feature branch and master branch. Both have remote branches and are in sync. Branch feature has some commits that master doesn't have and master are ahead of feature.

Keep the commits in feature but undo those commits and make a new commit in feature.

Find the most recent ancestor of feature and master git merge-base feature master

Say the commit is common1 Make a new branch to test if it's a real common ancestor git checkout feature git checkout -b featuretest common1 git log master..featuretest // You should see nothing and featuretest is not ahead of master

Then make feature to go back to that commit without changing history

= Method 1 = git checkout feature git checkout common1 . or git checkout -f c1 --. git commit -m "reset to common1" Now you can merge master into feature git merge –no-commit master

= Method 2 = / Revert commits one by one / Get commits from an old commit to HEAD git log common1..HEAD // or git rev-list common1..HEAD git revert –no-commit D git revert –no-commit C git revert –no-commit B … git commit -m "reset to common1"

= Method 3 = // Caution! You are rewriting history on feature branch at local and remote git checkout feature git reset –hard common1 git push -f

24.12.15. Find branches merged into a branch

# List local branches merged into master
# Use `-a` to show both local and remote branches
# Use `-r` to show only remote branches
git branch --merged master

# List local branches merged into HEAD (i.e. tip of current branch)
git branch --merged

# List branches that have not been merged
git branch --no-merged

24.12.16. Pull from remote/master to master branch which is not the current branch

  • git fetch origin master:master
    • Currently at local branch dev and need to pull remote/master to local master branch
    • This does not work if current branch is local master
    • Don't use git pull origin master as it pulls from origin/master to current local dev branch
    • update all origin/branchname branches
    • update all remote branches for all remotes

24.12.17. Orphan Branch with no parent

github:gh-pages

24.12.18. git:upstream

24.13. Merge

  • By default without any options, merge will not create merge commits when fast-forward is possible
  • --no-ff
    Default (--ff)
    create a merge commit only if the merge cannot fast forward
    --ff-only
    Merge when fast-forward is possible, otherwise refuse to merge
    --no-ff
    create a merge commit in all cases
    --no-commit
    stop just before creating a merge commit. There is no stop if fast-forward is possible.
    • Fix the conflict and git add files then normal commit
  • Resolve conflict
    • Conflict markers
      • <<<<<<<
      • =======
      • >>>>>>>
    • git add .
    • git commit -m "" or git merge --continue
  • git merge --abort
    Combine all integrated changes into a single commit instead of preserving them as individual commits. Need to make a normal commit after
    --squash

24.13.1. Merge just one file

Merge file f of branch B's HEAD into f of branch A~ 

git checkout A
git checkout B f

# --patch option makes it interactive
# if f on B does not exist yet, omit --patch, otherwise, you'll get a "No Change" message
# B could be any commit

24.14. merge-base 

# Return the most recent ancestor of 2 branches
git merge-base branch2 branch3

# Return the best common ancestor(s) between 2 commits

# If the return is identical to last-commit-feature-branch, then last-commit-feature-branch is in previous commit before last-commit-master-branch
git merge-base <last-commit-master-branch> <last-commit-feature-branch>

# Let's say master branch is tagged every time a deployment to live happens, let's say a tag is called `latest`, find that commit on master

git show latest

# copy this hash, let's assume it's last-commit-master-branch

# you have a commit abc, you want to find out if it is already deployed

git merge-base last-commit-master-branch abc
# eq. to
git merge-base master abc
git merge-base $(git log -1 --pretty=format:"%h" master) abc

# If it returns abc, then abc is deployed

# To get last commit of master branch
git log -1 master

24.15. Rebase

24.15.1. Rebase backup and revert

# rebase saves your starting point to ORIG_HEAD so this is usually as simple as:

git reset --hard ORIG_HEAD

# However, the reset, rebase and merge all save your original HEAD pointer into ORIG_HEAD so, if you've done any of those commands since the rebase you're trying to undo then you'll have to use the reflog.

# Referrence: https://stackoverflow.com/questions/134882/undoing-a-git-rebase

  • Old method 

    # Before doing rebase, tag the head of the current branch and easily reset to that after rebase if there's anything wrong

# If it shows fatal error, that's good! LILIBACKUP is not used
git show LILIBACKUP

# Create a tag to point to the current head commit
git tag LILIBACKUP

# do git rebase

# If there's anything wrong after git rebase is done, reset back to it
git reset --hard LILIBACKUP

# Delete the local LILIBACKUP tag
git tag -d LILIBACKUP

24.15.2. Rebase master onto feature branch

git checkout master
git pull --rebase
# or just git pull if you're sure local master can be fast forwarded
git checkout feature
# make sure local feature is up-to-date with remote feature
git pull
# make some changes and git commit

# find files that are different between local feature and local master
# refer to git:branch:file difference
git diff --name-status master..feature

# A status :: feature branch has a new file that master does not exist

git rebase master
# git status to review conflicts if exist. After resolving,
git add .
git rebase --continue

# there may be several rounds of continue
# if you see working tree is clean after `git status` and continue can't continue:
git rebase --skip
git status
# review and resolve conflicts as above
git rebase --continue

# now your commits in feature are applied on top of the master tip

# See file difference between master and the feature branch, these should list all changed files in feature branch
git diff --name-status master..feature

# The feature branch's new commits are on top of the master branch tip
git log

# check file difference between local feature and master to see if that's the change you want it
# you may need to manually make a commit on local feature to fix the rebase
# git add . && git commit -m "fix rebasing"

# you can merge commits into one commit on feature branch to further clean up your feature branch commits
git checkout master
git merge feature

The result is feature and master are exactly the same: C2 -> C3 -> C4

24.15.3. Rebase master onto a feature branch that has merges from master

  • my-branch is based on master and it has its own commits, merges from master and conflict resolves from time to time
  • The result we want is: modify my-branch history so that there's only one commit on top of the master branch
    • Benefits of using git merge --squash
      • We don't have to resolve the conflicts we previously resolved
      • You can review all the changes previously made
    • Caveat of using git merge --squash
      • Changes are moved to working tree and waiting for you to index and commit
# After many merges from master to my-branch, now we want to rebase master onto my-branch:
git checkout my-branch
git branch -m my-branch-old
git checkout master
git checkout -b my-branch
git merge --squash my-branch-old
git commit
git branch --set-upstream-to origin/my-branch
git push -f

You want to see what changes are made for branch-for-cr 

git checkout branch-for-cr
git pull --rebase=i
# This is to make sure local branch-for-cr is truly the same as remote because remote history might be changed
# - If the rebase prompts to select/pick commits, pick all commits (by doing nothing) `:wq`
# - `git rebase --skip` for all of them
# Now local branch-for-cr is exactly the same as remote branch-for-cr
git checkout master && git pull
git checkout -b t-branch-for-cr
git merge --squash branch-for-cr
git add . && git commit -m "for CR"
# Review the changes in the new commit

24.15.4. Rebase a base feature branch onto another feature branch

  • feature-b depends on feature-a and it should contain all changes from feature-a
    • feature-a commit history is changed constantly
      • It's always on top of master because master is continuously rebased onto feature-a
      • feature-a might git merge --squash into master when feature-a is merged into master. See 24.15.3
    • feature-b is always on top feature-a
  • See https://softwareengineering.stackexchange.com/a/351790/294454
# Start feature-b
git checkout feature_a
git checkout -b feature_b

# feature_b has some updates
git add . && git commit -m "feature_b update"

# feature_a now has more commits

# feature_a may rebase master
git checkout feature_a
git rebase master

# feature_a is merged to master

# feature_a is squash merged to master
git checkout feature_a
git branch -m feature_a_old
git checkout master
git checkout -b feature_a
git merge --squash feature_a_old
git add . && git commit -m "Merge feature_a into master"
git branch --set-upstream-to origin/feature_a
git push -f

# Every now and then, we need to rebase feature_a onto feature_b
# Pull changes for feature_a (history might change)
git checkout feature_a
git pull --rebase=i
# This is to make sure local feature_a is truly the same as remote because remote history might be changed
# - If the rebase prompts to select/pick commits, pick all commits (by doing nothing) `:wq`
# - `git rebase --skip` for all of them
# Now local feature_a is exactly the same as remote feature_a

git checkout feature_b
git rebase feature_a --interactive
# `:wq` to pick every commit

# Scenario #1:
# feature_b before rebasing feature_a: (M1-M2-A1-A2)-B1-B2
# feature_a before: M1-M2-A1-A2
# feature_a after: M1-M2-M3-A1'-A2'-A3
git rebase feature_a --interactive
# Find the first commit of feature_b, which is B1
# `:wq` to pick all commits to replay
# - Any commits that have conflict (from A1 to A2 on feature_b) and possibly any B1-B2
# - Pick all of them
# `git rebase --skip` for any commits that before B1. Like (A1-A2)
# Don't skip any B1-B2
# feature_b after rebasing feature_a: M1-M2-M3-A1'-A2'-A3-B1'-B2'
# Summary: mark any commits before B1 (M1-M2-A1-A2) to skip in case `git rebase` prompts

# Scenario #2:
# feature_b before rebasing feature_a: (M1-M2-A1-A2)-B1-B2
# feature_a before: M1-M2-A1-A2
# feature_a after: M1-M2-M3-A4
git rebase feature_a -i
# Find the first commit of feature_b, which is B1
# `:wq` to pick all commits to replay
# `git rebase --skip` for any commits that are before B1. Like (A1-A2)
# Don't skip any B1 and onwards
# feature_b after rebasing feature_a: M1-M2-M3-A4-B1'-B2'

# Finally, as soon as feature-a is merged into master
git checkout master
git pull
git checkout feature_b
git rebase --onto master feature_a feature_b

24.15.5. Interactive: Commands p(pick), s(quash), e(edit)

  • :cq to quit Vim without saving. Which means abort the current interactive rebase

24.15.6. Branch preference

git checkout feature-a
# favor feature-a changes over master
git rebase -Xtheirs master

# favor master changes over feature-a
git rebase -Xours master

# -Xxxx is reversed in git merge!

24.15.7. Squash commits into one commit

Branch feature has 4 commits ahead of branch master
m1 > c1 > c2 > c3 > c4
(no term)
Combine c2, c3 and c4 (c1 is not included!) to c5 and assign a different commit message
# interactive rebase for commits NEWer than c1
git rebase -i c1
# open up VIM, keep the first line unchanged pick c2 and change pick to `s` or `squash` for all other commits
pick c2
s c3
s c4

# `ESC` and `:wq` to save, VIM returns that what will happen. Comment lines with # and give a new commit message without #

# a new SHA commit c5 is created with a commit msg that is specified

# `ESC` and `:wq` to save. Done!

Commands other than p(ick), s(quash):

e(dit)
make file changes, git add myfile and to continue to next command or commit git commit --amend

24.15.8. Reorder commits

git checkout branchA

# New commits including a
git rebase -i a^

# Inital order
# p a
# p b
# p c
# p d
# p e
# p f
# p g

# Reorder

# p c ...
# p a ...
# s d ...
# s e ...
# s g ...
# p b ...
# s f

24.15.9. Split a commit

  • git rebase -i c123^
  • Assign e or edit to the commit c123 after git rebase is run, Esc :wq to start the rebase process
  • git reset HEAD^
    • git status shows Last command done
      • edit c123 some commit message
    • git add . && git commit -m "message"
  • git rebase --continue

24.16. Cherry pick

24.16.1. Example

Cherry pick commit f from Feature to master 

a - b - c - d   Master
     \
   e - f - g Feature

git checkout master
git cherry-pick f

After 

a - b - c - d - f'   Master
  \
    e - f - g Feature

Let's say master has a new commit h 

a - b - c - d - f' - h  Master
  \
    e - f - g Feature

After git checkout feature && git rebase master, it becomes
a - b - c - d - f' - h  Master
  \
    c - d - f' - h - e - g Feature

The original f is dropped

24.17. Show child commits for merge commit

After merge or rebase happened, a merge commit is created. git status gives :: C0 > C1 > merge commit abc

See what this merge commit abc consists of. Take a look at the git status of abc, you will find Merge: C1 xyz

Show file names only :: git diff –name-only C1..abc Show commits in a merge commit :: git log C1..abc Show file names in each commit in a merge commit :: git show –name-only C1..abc Show changes of a file that is caused by the merge commit :: git diff C1..abc – path/to/file

24.18. Fork

Sync a fork with original repo Add a remote: upstream 

git clone git@github.com:YOUR-USERNAME/YOUR-FORKED-REPO.git
cd into/cloned/fork-repo
git remote -v
git remote add upstream git://github.com/ORIGINAL-DEV-USERNAME/REPO-YOU-FORKED-FROM.git
git remote -v
git fetch upstream
git pull upstream master

24.19. Symlink

  • core.symlinks defaults to true
    • Git detect if the current filesystem supports symlink (FAT does not support) during git clone and git init
      • if not, it will set it to false on the project folder
  • When it is false, the symlinks will be created as plain text file. To pull from remote again and make those plain text files to be symlinks:
    • Keep core.symlinks=false, create 3 and remove tracking for those symlink files
      • git update-index --assume-unchanged symlink1 using git:update-index

      • To find all symlink files but are not created as symlinks on Windows: 

        git ls-files -s | awk '/120000/{print $4}'
      • Change it to true for the current repo
        • git config core.symlinks true
      • and then git pull

24.20. Remove tracking for commited files

  • A file is commited but you don't want to track it anymore in the future
    • git update-index --assume-unchanged path/to/file.txt
    • Remember the file will still be in Git but future clone won't grab the file
  • If you change your mind and want to track it again
    • git update-index --no-assume-unchanged path/to/file.txt
  • Don't track a folder of files which are already commited
    • First try to run this because this will work with file names with space
      • git ls-files -z myFolderToIgnore/ | xargs -0 git update-index --assume-unchanged
      • Path is relative
      • If you encounter error xargs: git: bad file number, run this
        • git ls-files -- myFolderToIgnore/ | xargs -l git update-index --assume-unchanged
  • List all assumed unchanged files
    • git ls-files -v | grep '^[[[[:lower:]]]]'
  • If repo has new commits for locally assumed unchanged files, Git will prevent from merging/pulling
    • set it to --no-assume-unchanged, stash the change or remember the change, merge/pull, and change to assumed-unchanged, and apply the previous change

24.21. Remove a commited file from file but leave in filesystem

Delete a file from Git (top of the current branch) but leave the file in filesystem The file becomes Untracked File 

git rm --cached file.txt

# remove a folder
git rm -r --cached .emacs.d/

Add file to local or global ignore so that the file won't appear as Untracked File and prevent from staging the file when git add .

24.22. Remove large files and folders

Reference
https://docs.github.com/en/github/authenticating-to-github/removing-sensitive-data-from-a-repository
git gc
# Get the top 10 biggest files. It will take about a minute for it to run
git rev-list --objects --all | grep "$(git verify-pack -v .git/objects/pack/*.idx | sort -k 3 -n | tail -10 | awk '{print$1}')"

# path/to/bigfile1, path/to/bigfile2
# filter-branch runs on the current branch and filter through from top (HEAD) to oldest
git filter-branch --index-filter 'git rm --cached --ignore-unmatch path/to/bigfile1' HEAD
# repeat for every file

# To remove a directory
git filter-branch --index-filter 'git rm -r --cached --ignore-unmatch path/to/folder1' HEAD

# To remove files and subdirectories (except hidden files) of a directory
git filter-branch --index-filter 'git rm -r --cached --ignore-unmatch sites/default/files/*' HEAD

# filter-branch creates backups of your original refs namespaced under ~refs/original~
# Run this to delete the backed up refs, allowing the large objects to be garbage collected
git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d

# if all big files are in one branch only, just delete the branch will remove all the references
# git branch -D branch-name
# Prune all reflog references from the branch on back.
# git reflog expire --expire=now branch-name

# Prune all reflog references from now to the first commit
git reflog expire --expire=now --all

# Repack the repo by running gc and pruning old objects
git gc --prune=now

# Push changes back to remote repo. All branches (refs under ~refs/heads~)
# I found I can push one branch only
git push --all --force

# Make sure tags are current and pushed to remote
git push --tags --force
–index-filter
modify a repo's staging
–cached
remove a file from the index not the disk
–ignore-unmatch
prevent git rm from failing if the file isn't there
HEAD
to remove from the starts

Say git filter-branch is run once, and the backup refs/original is not deleted, the subsequent filter-branch will not run Add -f to force it to run and overwrite refs/original 

git filter-branch -f --index-filter [...]
  • git filter-branch [...] HEAD
    • just shrinks the current branch and leave the other branches intact.
    • Say feature branch is a copy of master, then run filter-branch on feature. master branch and others are not affected. But now feature branch diverged from master.
  • The get-top-10-biggest-files command returns the same files after filter-branch and the cleanup mentioned above are done. Because it's just one branch that you cleaned up, big files are still in other branches.
    • You won't notice a big change in total git:repo size, as it's just one branch that had the big files removed.
  • I didn't try git push --force to update a real remote branch.
    • Let's say the repo is at /path/to/repo, and you duplicate master branch to shrinksize branch. Run filter-branch and the cleanup.

    • Then you create another folder /path/to/duplicaterepo, and clone only the shrinksize branch from the original repo 

      cd /path/to
mkdir duplicaterepo
cd duplicaterepo
git clone file:///path/to/repo --branch shrinksize --single-branch
cd repo
du -sh .git/objects
# push branch shrinksize to another remote as master branch
git remote add anotherremote ...
git push anotherremote shrinksize:master

24.23. Remove one file from all remote branches

git filter-branch --force --index-filter \
"git rm --cached --ignore-unmatch ./path/to/file" \
--prune-empty --tag-name-filter cat -- --all

# --prune-empty :: some filters will generate empty commits that leave the tree untouched. This instructs git-filter-branch to remove such commits if they have exactly one or zero non-pruned parents; merge commits will therefore remain intact

# --tag-name-filter <command> :: This is the filter for rewriting tag names. The original tags are not deleted, but can be overwrriten; use "--tag-name-filter cat" to simply update the tags.

# wildcards can be used e.g. ./path/to/*.xml
# To remove a directory: git rm --cached --ignore-unmatch -r ./path/to/folder

# Add the file to .gitignore
echo "path/to/file" >> .gitignore
git add .gitignore
git commit -m "add path/to/file to .gitignore"

git push origin --force --all --dry-run
git push origin --force --all
git push origin --force --tags --dry-run
git push origin --force --tags

24.24. log - Show Commit History

Syntax
git log [<options>] [<revision range>] [[--] <path>…​]
(no term)

Commit formatting 

git log --oneline
git log --all --decorate --oneline --graph
(no term)
Options
--decorate[=short|full|auto|no]
default is short, just displays the commit hash
(no term)
--pretty[=<format>], --format=<format>
medium
default
oneline
<hash> <title-line>
(no term)
Placeholders
%as
author date YYYY-MM-DD
%s
subject
(no term)
-- path/to/file/or/director
  • git log origin/master -- path/to/a/file
(no term)
show commits on origin/master which modify a file
  • git log -- path/to/a/file
(no term)
--branches[=<pattern>]
  • The pattern should be any local branches (not remote branches) which are any of .git/refs/heads/*
  • If pattern is empty, commits of all local branches are included
(no term)
-all
  • All commits of all local and remote branches are included which are all of .git/refs/*
(no term)
See git:log:commit difference
git log -3
Last 3 commits
  • git log -3 eq. git log -n 3 git log --max-count=3
git log commit1..commit2 --format=oneline | wc -l
Count commits in a commit range, returns 1
git log --grep=<pattern>
Search commit message
%s
subject (commit message)
(no term)

Show commits for multiple repos between start/end dates 

#!/bin/bash

TMPLOG=/mnt/e/project-logs.csv
touch $TMPLOG
echo "" > $TMPLOG
gitrepos=( '/mnt/e/li/repo1' '/mnt/e/li/repo2' )

for i in "${gitrepos[@]}"; do
   if [ -d $i ] && [ -d $i/.git ]; then
  cd $i
  git log --since='last 2 months' --pretty=format:"$i,%ai,%s" >> $TMPLOG
  # After and including May 1, 2018
  # git log --after='2018-5-1'
  echo "" >> $TMPLOG
  cd ..
   fi;
done
Show commits of a specific branch
git log origin/master
(no term)
Show an author's commits
  • Current branch
    • git log --author="Li Li" --before="2018-12-01" --after="2018-11-01"
  • All branches include remote and local
    • git log --author="Li Li" --after=yesterday --all
  • --author
    • may use email
    • --author='Li*'
  • match the regular expression limiting patterns without regard to letter case
  • --since=<date>, --after=<date>
    Today
    --after=midnight
    yesterday
    --after=yesterday
    (no term)
    Commit date
  • a commit's creation date
    • git log --author='Li Li' --format='%as %s' | grep -E ^2022-08-18
    • Today
Show an author's commits of a specifc branch
git log origin/master --author="Li Li"
Show an auhtor's commits by number of files changed, insertion and deletion
git log --author="Li Li" --oneline --shortstat
Show an author's commits by lines added/deleted for each file
git log --author="Li Li" --pretty=tformat: --numstat
(no term)

Exclude sub folders and files or include 

# -- <path> :: limit commit history by path
git log -- . ":(exclude)subfolder"

# case-insensitive
git log -- . ":(exclude,icase)wp-content/plugins"

# exclude multiple
git log -- . ":(exclude,icase)wp-content/plugins" ":(exclude,icase).idea"

# include only
git log -- ":(icase)wp-content/themes" ":(icase)wp-content/plugins"

# include and exclude
git log -- ":(icase)wp-content/themes" ":(exclude,icase)wp-content/themes/abc"
(no term)

Show an author's commits by total lines added/deleted 

git log --author="Li Li" --pretty=tformat: --numstat | awk '{ add += $1; subs += $2; loc += $1 - $2 } END { printf "added lines: %s, removed lines: %s, total lines: %s\n", add, subs, loc }' -

24.24.1. git-when-merged

# Find the merge commit that first brought the commit into the master branch
# -l :: show the merge commit's message
git-when-merged -l a-commit-hash master

# -r :: follow the merge commit recursively
git-when-merged -lr a-commit-hash master

24.25. Ignore, Local/Global Ignore, Show Ignored Files

  • https://www.gitignore.io/
  • Refer to git:update-index
  • git status -u
  • Add file or directory paths to
    • .git/info/exclude/ (will not be committed)
    • .gitignore file (will be commited)
    • Create touch .gitignore
  • ~\.gitignore_global and run git config --global core.excludesfile ~/.gitignore_global
  • git config --list --show-origin
    • Sometimes, Windows is set up for all applications to use user profile folder at a different location, e.g. phpStorm
      • Change excludesfile to a folder location that you know under [core] after git config --global -e (edit the global config file)
      • Or try this git config --global core.excludesfile K:/.gitignore_global
  • git status --ignored
  • git check-ignore -v path/to/filename
  • After 1.8.2, ** can be used to mean zero or more sub-directories (relative to the place where .gitignore is)
    • bin/ ignores any folder named bin
    • /bin/ ignores one folder named bin relative to the .gitignore file

    • However, if there're more than 1 folder, wp-content/uploads/ only ignore /wp-content/uploads. Use **/wp-content/uploads/ to ignore any folder that has path wp-content/uploads 

      **/.DS_Store
/main/**/bin/

  • Negate. An optional prefix ! which negates the pattern; any matching file excluded by a previous pattern will become included again. If a negated pattern matches, this will override lower precedence patterns sources. 

    # Ignore everything
*

# But don't ignore these files...
!.gitignore
!script.pl
!template.latex
# etc...

# ...even if they are in subdirectories. (don't ignore first level subdirectories)
# without this line, script.pl will only be included in the root directory
!*/

  • Ignore a folder but keep files in subdirectories, create .gitignore in that folder 

    # ignore everything
*

# add gitignore
!.gitignore

# keep sql files
!*.sql

# keep subdirectories
!*/

  • e.g. ignore a directory site[123] 

    site\[123\]
  • *~
  • Ignore a commited file but leave it in filesystem
    • Add the file to .gitignore
    • git rm --cached file.txt
    • git rm -r --cahced .emacs.d/
    • This way the file won't appear as Untracked File and prevent from staging the file when git add .
  • https://www.atlassian.com/git/tutorials/gitignore

24.25.1. WordPress .gitignore git:gitignore:wordpress

*~
cgi-bin/
.well-known/
.DS_Store
.svn
.cvs
*.bak
*.swp
Thumbs.db
wp-config.php
wp-content/uploads/
wp-content/blogs.dir/
wp-content/upgrade/
wp-content/backup-db/
wp-content/advanced-cache.php
wp-content/wp-cache-config.php
wp-content/w3tc-config/*
wp-content/cache/*
wp-content/cache/supercache/*
*.log
error_log
wp-content/plugins/error_log
wp-content/cache/
wp-content/backups/
sitemap.xml
sitemap.xml.gz
sql-admin/
cs-devops/

24.25.2. NodeJS, NPM .gitignore

node_modules
npm-debug.log

24.26. Customize Settings per folder

For a single repository, attributes should be placed in $GIT_DIR/info/attributes file. $GIT_DIR is the repository. But this file won't be commited.

If you want to commit, .gitattributes file in the root $GIT_DIR or any sub folders. 

# Syntax
# pattern attr1 attr2 ...
*.txt text*.js text*.html text*.md text*.json text*.svg text

# text is an attribute without value. `Set [attr] to true`

*.pdf -text

# text is again an attribute without value but prefixed with -. `Unset [attr] or Set [attr] to false`

*.txt text eol=lf

# Set text to true and Set eol to a value lf

24.27. Git Windows

gitk
GUI
start path/to/filename
to tell Windows to open a file
(no term)
edit a file using vi vi afile
Run tree command
cmd //c tree
(no term)
At Git Bash, open the current folder in Windows Explorer explorer .

24.27.1. Line Feed, Long Path

Always don't do any conversion when files are added to local index or codes are checked out to local filesystem
git config --global core.autocrlf false
(no term)
Do git clone again
(no term)
May encouter file name too long or pathname too long error.
Change the git setting
git config --system core.longpaths true

24.27.2. Special characters in filename

You can git clone in Linux, tar it and untar it using 7zip on Windows. 7zip will change the filenames to fit in Windows.

24.27.3. File name case-insensitive

Git has 2 file names the same except casing. After clone to Windows, you will find Changes not staged for commit.

If the files are not important and you are not going to modify them on Windows, you can tell your local Windows Git not to checkout those files. 

git config core.sparsecheckout true
echo '*' >.git/info/sparse-checkout
echo '!unwanted_dir/unwanted_filename' >>.git/info/sparse-checkout
# I found just unwanted_dir/ is not enought, have to repeat each problematic files returned by `git status`
git read-tree --reset -u HEAD
git status

If the files are important, you can use Git in Bash on Windows, it will checkout files with case-sensitive names. But all other Windows applications including Git Windows and phpStorm will report wrong Git info. Keep using Bash on Windows to Git.

It's better to fix the file names on Linux, commit and push and work on local..

24.28. Credential Manager, Multiple GitHub accounts and https

  • You manage 2 GitHub accounts
  • git clone does not require credentials for https
  • When push to the remote which is under GitHub account A, Windows asks for credentials
  • You can add A to any repo as a collaborator
    • Any push will be made by account A
  • You have to delete 2 GitHub credentials from WCSP and then push commits to account B's repo and then use account B's credential if you want to completely switch account
  • If you want to switch between multiple GitHub accounts frequently, use SSH instead of https
    • SSH might be blocked in public internet networks though
  • git config --global crendential.helper wincred

  • On Windows, disable Credential Manager and turn off OpenSSH popup to directly type password in shell. Still need to type in password for all https repo. Run in PowerShell with admin 

    git config --system --unset credential.helper
git config --edit --global

    Insert to global ~/.gitconfig 

    [core]
    askpass =

  • Always ask for password for a local repo 

    # this does not seem to work
# git config credential.helper cache

# single command
git -c credential.helper= push
  • We can setup Personal Access Token instead of using the account password to access GitHub
    • Only HTTPS repo (not SSH) can use personal access token
    • to create a new token. It only shows one
    • git config --global credential.helper wincred or you may need to git config --system --unset credential.helper
    • Remove [core] askpass = in ~/.gitconfig
    • maybe delete old record git:https://your@email.com@github.com

24.29. clone

24.29.1. Clone to an empty folder

cd ~/expressjs
# add .git at the end
# --bare not to create a directory
git clone --bare https://githublink/express.git .git
git config --bool core.bare false
git reset --hard

24.29.2. Clone or Pull to existing repository

24.29.2.1. Existing repo is not git initialized

The existing repo has some other files that need to keep but not version controlled. In the existing repo run 

git init
git remote add origin https://github.com/username/reponame.git
git fetch --all
git reset --hard origin/master
# dump all untracked changes and checkout remote master branch
24.29.2.2. Existing repo is git initialized

Pull from remote and put changes to staged without commit (ready to commit) 

git pull --no-rebase --squash -Xtheirs origin master
# origin can be a different upstream in URL format
# git://github.com/pantheon-systems/drops-7.git
# the remote that's pulled from usually has the same codebase as your current local repo

24.29.3. Clone a subdirectory --depth

Just clone examples/shopping-cart directory from branch dev 

git init <repo>
cd <repo>
git remote add origin https://github.com/vuejs/vuex.git
git config core.sparsecheckout true
echo "examples/shopping-cart/*" >> .git/info/sparse-checkout
git pull --depth=1 origin master
# Result is ~./<repo>/examples/shopping-cart~

# Grab code only
# `--depth` implies `--single-branch`
git clone --depth=1 --branch=master git://someserver/somerepo dirformynewrepo
rm -rf ./dirformynewrepo/.git

24.29.4. Clone a branch only --single-branch -b, --branch

git clone --single-branch -b <branchname> https://github.com/...

24.30. Track local branch with a remote branch 

git branch --set-upstream-to=origin/master
# Old
# git branch --set-upstream master origin/master

List all local branches' upstream remotes 

git branch -vv

24.31. List all remote repo

# List all remote URL's 
git remote -v

24.32. Maintenance, Repo Size git:repo size

  • After a branch is deleted, commits that belong to no branch still exist in database
    See all corruptions
    git fsck --full and it shows the following types (not all types)
    Commits that belong to no branch and no tag is attached (dangling commits)
    git fsck --unreachable
    Tags that don't point a commit (dangling tag)
    git fsck --no-reflogs
    Remove all tags in local repo
    refer to git:tag
    Mark reflog entries of all types as expired and prune them
    git reflog expire --expire=now --all
    Prune garbage
    git gc --prune=now or git prune
    Dry run prune
    git prune --dry-run --verbose
  • https://git-scm.com/book/en/v2/Git-Internals-Maintenance-and-Data-Recovery

24.32.1. Repo size

git gc
git count-objects -vH

# I found this better
du -hs .git/objects

24.33. Number of tracked files

Doesn't include directories. Only files git ls-files | wc -l

24.34. GitHub Push to New Repo with local repo

  • Create a new repo on Github without creating any .gitignore, README nor license
  • On local, git init, add .gitignore, git add . and check if files are ignored, git commit -m "1st commit"
  • Add a remote
    • git remote add origin ssh_or_github_https_link_with_.git_at_the_end
  • List all remotes git remote -v
  • Remove a remote 'origin' git remote rm origin
  • Local branch is not set to sync with remote until you do the first push with '-u'

git push -u origin master

Finally check cat .git/config to see if local branch "master" has setup a remote 

[branch "master"]
    remote = origin
    merge = refs/heads/master

You may encounter 403 Forbidden and HTTP request failed errors. Try to use SSH or modify the HTTPS url 

# Change HTTPS remote URL in `.git/config` to this
https://[GITHUB_USERNAME]@github.com/[GITHUB_USERNAME]/[REPO_NAME].git

# Because some network blocks the default HTTPS URL
https://github.com/[GITHUB_USERNAME]/[REPO_NAME].git

24.35. Add a remote, remove a remote and its branches, change upstream

git remote add origin https://...
git remote -v

# remove a remote
git remote rm origin

# rename a remote
git remote rename origin li

# change upstream to origin/master for local branch master
git branch master --set-upstream-to origin/master

Change remote from https to ssh. You need to add public key to GitHub first 

git remote -v

# origin  https://github.com/USERNAME/REPOSITORY.git (fetch)
# origin  https://github.com/USERNAME/REPOSITORY.git (push)

git remote set-url origin git@github.com:USERNAME/REPOSITORY.git

24.36. Push to 2 remotes

An existing remote: origin 

git remote add origin ssh_or_github_https_link_with_.git_at_the_end

# check remotes
git remote -v

cat .git/config
# [remote "origin"]
#   url=ssh://.../repo.git
#   fetch= +refs/heads/*:refs/remotes/origin/*

Method 1: Configure Origin as a Multi-Remote Destination Add another push URL for origin within .git/config Commits will be pushed to both remote destinations automatically on git push origin, fetch will still fetch from the first URL 

git remote set-url --add --push origin https://original-remote.com/myproject.git
git remote set-url --add --push origin https://second-remote.com/myproject.git

[remote "origin"]
  # default are the following 2 lines. If pushurl does not exist, url is used for pushing
  url = https://original-remote.com/myproject.git
  fetch = +refs/heads/*:refs/remotes/origin/*
  # The following 2 lines are added. Sequence does not matter
  pushurl = https://original-remote.com/myproject.git
  pushurl = https://second-remote.com/myproject.git

git remote show origin lists the Fetch and Push URL's 

* remote origin
  Fetch URL: https://original-remote.com/myproject.git
  Push URL: https://original-remote.com/myproject.git
  Push URL: https://second-remote.com/myproject.git
  HEAD branch: release
  Remote branches:
	branch-1                                        tracked
	branch-2                                        tracked

Method 2: Add another remote github 

git remote add github git@github.com:username/reponame.git
# or a https directly from github
# Push and track the current local branch master
# this will change the trakcing remote from origin to github
# git push -u github master

# just push the current branch to another remote manually
git push github

24.37. Tag

  • A tag points to a commit and the commit may be on multiple branches
# list all tags
git tag

# search tags
git tag -l "v1.8.5*"

# reverse order
git tag -l --sort=-version:refname
# or try --sort=-refname

# Make annotated tag and point to the commit at HEAD
git tag -a v1.4 -m "tagging message"

# Show a tag
git show mytag

# see which commit a tag points to
git rev-list -n 1 mytag

# make lightweigth tag and assign the tag to the commit at HEAD
# lightweight tag doens't show anything in `git show`
git tag v1.4-lw

# Attach an annotated tag at a specific commit
git tag -a v1.2 0fceb02

# this only pushes the tag not the commits
git push origin [tagname]
# or push all local repo tags to remote repo
git push origin --tags

git describe --long --match 'live*'
# live-10-g7digits
# HEAD is now 10 commits ahead of tag live

# In order to move the tag to point to another commit, have to delete and create
git tag -d latest
git push origin :refs/tags/latest
git tag -a latest -m "some note"
git push origin latest

# delete all tags in local repo
git tag -d $(git tag -l)

# if got error `argument list too long`, run several times of this
git tag -d $(git tag -l | head -n 100)

# fetch all tags from remote repo
git fetch --all --tags

# checkout a tag
git checkout tags/v1.0

# create a new branch at a certain tag
git checkout tags/v1.0 -b mynewbranch-v1

# Delete a tag
git tag -d <tag_name>

# Delete a tag in remote
git push --delete origin <tag_name>

24.38. Submodule

// create a folder DbConnector in parent project directory root folder
cd /parent
git submodule add https://github.com/chaconinc/DbConnector
// change the subfolder name from DbConnector to dbconnector
// git submodule add https://github.com/chaconinc/DbConnector dbconnector

24.39. Git-flow

24.39.1. Case Study

  • FeatureA merged to LSR > FeatureA merged to Master > Hotfixes related to FeatureA, FeatureX and FeatureY on Master (HOTFIXes_A-X-Y) > Master merged to FeatureB > FeatureB merged to LSR > conflicts
    • HOTFIXes_A-X-Y on Master cause the conflicts
      Some valid reasons
      master is behind and diverged from LSR. LSR might fix some issues of FeatureA in other commits that are not yet merged to master or the way LSR fixes the issues of FeatureA differently.
    • Developers of FeatureB (and all other Feature branches actively being merged to LSR) DO NOT know how to resolve conflicts when they merge master to FeatureB and merge FeatuerB to LSR. FeatureB does not have code changes for FeatureA, X and Y.
    • My suggestion
      1. HOTFIXes_A-X-Y should be merged back to LSR. It will save a lot of developers' time and effort on resolving conflicts. These developers include ones who work on branches (e.g. FeatureB) which have no code change for FeatureA and FeatureX and FeatureY
        • Resolving conflicts when merging HOTFIXes_A-X-Y to LSR, we may totally ignore any hotfix that is done on master because LSR might already have fixes.
      2. HOTFIXes_A-X-Y should be merged back to any future branches that have not been and will be merged to master and have changes for FeatureA, X and Y.
    • My comments
      • At the end of the day, LSR should be merged to master. Relying on merging individual feature branches to master is dangerous. In this example, FeatureA merges to LSR without a problem but merging to master requires hotfixes on master. These hotfixes should be merged to any ongoing related branches including LSR.

24.40. GitHub

24.40.1. API

REST API v3
https://developer.github.com/v3/
(no term)

Get repo size 

# for public repo, auth is not required
curl https://api.github.com/repos/:owner/:reponame | grep size

# for private repo
curl -u ownername:password https://api.github.com/repos/:owner/:reponame | grep size

# sort repos of user:alibaba by number of stars
https://github.com/search?q=user%3Aalibaba+&s=stars&type=Repositories
(no term)
Webhook

24.40.2. GitHub Pages

  • Available in
    • public repos with free GitHub accounts
    • private repos with Pro, Team, Enterprise Cloud, Enterprise Server accounts
  • add /docs folder to your repository on master
    • Or create an orphan branch called gh-pages
    • When branch gh-pages and /docs in master branch not exist, just click on Launch automatic page generator and it will create the gh-pages branch with Jekyll files
    • For User or Organization Site, create repo levonlee.github.io and add index.html and push to repo
  • enable GitHub Pages in the repository setting
  • docs.abc.com with CNAME YOUR-GITHUB-USERNAME.github.io 

git checkout --orphan gh-pages
git rm -rf .
git commit -m "GitHub Pages"
24.40.2.1. SVG, View HTML

http://rawgit.com/

Image files other than SVG can be served directly from GitHub using Raw Repo Path.

Blob Repo Path
https://github.com/levonlee/cssSandbox/blob/master/assets/img/sprite.svg
(no term)
Raw Repo Path
  • https://github.com/levonlee/cssSandbox/raw/master/assets/img/300x250x1.jpg
  • https://raw.githubusercontent.com/levonlee/cssSandbox/master/assets/img/300x250x1.jpg

For svg and html, use rawgit

RawGit Path
/levonlee/reponame/master/folder1/s1.svg
RawGit Path for Gist
levonlee[long-gist-id]/raw/index.html
Whole Path
https://rawgit.com/levonlee/cssSandbox/master/{dir}/index.html
Whole Path for Gist
https://gist.githubusercontent.com/levonlee/[long-gist-id]/raw/index.html

Development (Traffic limited) https://rawgit.com/ + RawGit Path Production (No traffic limit, permanently cached, no url query param) https://cdn.rawgit.com/ + RawGit Path

RawGit is down.. Use raw.githack.com

24.40.2.2. Markdown

![Alt text](http://url/to/img.png)

24.40.3. Plan - Individual vs Team vs Enterprise

  • Convert an Individual Plan (IP) e.g. Pro to an organization with Team Plan (TP) or Enterprise Plan (EP), after conversion
    • Username of the original plan (OP) cannot be changed and will be carried on to TP
    • Email of OP cannot be used to login. Use assigned owner (other GitHub account) to manage the converted TP
    • Collaborators of all repos before IP to TP/EP conversion become members
  • IP can be a member/owner of any number of organizations
  • EP is created to manage policy and billing for multiple organizations
  • Organization
    Billing
    Charged by count of members and OC for private repo
    (no term)
    Webhook can be set on an organization
    (no term)

    Organization Permissions

    • Owners are Members
    • Admin permissions for all repos and manage billing

    • Approve or deny when a member requests OAuthe App access to org resources

      • Organization > Settings > Third-party access
      • Members
    • Create repo, teams, project boards
    • Can be made a team maintainer
    • Can be converted to outside collaborators (OC) and OC can never create repos nor the following mentioned

    • Org > Settings > Member privileges

      Base permissions
      every member has one of these permission levels for all org repos
      None
      clone and pull public repos
      Read
      clone and pull all repos
      Write
      clone, pull and push all repos
      Admin
      clone, pull, push and add new collaborators to all repos
      Repo creation
      a member who creates a repo will have Admin Repo Permission Level
      • Public and private repos
      • Private
      • Disabled
      Billing Managers
      view and edit billing
      Outside Collaborator (OC)
      OC is not a member and if he's assigned to a private repo, it will count as one seat in billing
    • Can't
      • Create repo
      • Create teams
      • See all org members and teams
      • @mention any visible team
      • Be a team maintainer
    • Can have all Repo Permissions
      • GitHub Apps managers
    • Assign specific people to manage all GitHub Apps or just one
    Team
    • Team > Repositories
    • Assign repos for members in a team to manage, with Repo Permission Levels
      • Team > Teams > Nested teams

    • Child team can have only one parent team and it inherits the parent's access permissions

      • Members in child team will receive notifications when the parent team is @mentioned
      • Team > Members
    • can add and remove team members and create child teams
      • Team Pages
      • Team > Settings
    • Team Visibility
      Visible
      can be viewed and @mentioned by every Org member
      Secret
      only visible to members on the team and Org Owners
      • Can't have child teams and Can't be a child team
      • Good for external partners or clients
    Repo Permission Levels
    • Permissions can be given to members, OC and teams
    • Read - can pull
    • manage issues and pull requests without write access
    • Write - can push
    • manage repo without access to sensitive actions e.g. managing security or deleting a repo
    • Admin
    (no term)
    A member's permission
    The highest given in 3 categories
    Read, Write and Admin
    (no term)
    Permissions can be given in Team, Organization Permissions and Repo Permission Levels

24.40.4. Shortcut

Search
⌘ k

24.41. GitLab

24.41.1. Comment on a file of a commit

  • Comment can be made on a file of any line when the file is in a Merge Request (MR)
  • To make a comment for a line without the Merge Request context, we can
    • create an MR
    • Create an issue with a link to the line of the latest commit which changes the file
      Left sidebar
      Project > Repository > Files > Find file to open the file
      (no term)
      Click on History and go to the most recent commit of the file
      View file @commit-hash
      the default view only shows relevant lines of the commit, this opens the whole file so that any line can be referred to
      Go to the line and click on it, URL is changed, copy the URL
      this copies the location of the line of a specific commit
      Left sidebar
      Create an issue, paste the link and tag the relevant Dev as Assignee who will later gets notified and resolve the issue
    • Comment on the line of the latest commit which changes the line
      Left sidebar
      Project > Repository > Files > Find file to open the file
      (no term)
      Click on Blame and go to the line, click on the last commit which changes the line
      (no term)
      Comment on the line. Mention someone to make sure the developer is notified

24.41.2. CI/CD

  • Install a runner, register a project to use the runner
  • Create a project file .gitlab-ci.yml telling the runner what to do
    • GitLab Pages
24.41.2.1. Use a docker container as a runner

  • https://docs.gitlab.com/runner/install/docker.html 

    docker volume create gitlab-runner-config

docker run -d --name gitlab-runner --restart always \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v gitlab-runner-config:/etc/gitlab-runner \
    gitlab/gitlab-runner:latest

docker exec -it gitlab-runner bash

# inside, register
sudo gitlab-runner register

# How to register? https://docs.gitlab.com/runner/register/index.html#docker

# After registration, the config file is /etc/gitlab-runner/config.toml

# restart the container
docker restart gitlab-runner
24.41.2.2. .gitlab-ci.yml
  1. GitLab Pages
    • https://code.mortgageautomator.com/help/user/project/pages/getting_started/pages_from_scratch.md
    • public is the directory serves the static website

    • Option 1: just use the public directory, make a job called pages, specify the public directory 

      pages:
  script:
    - echo "Anything in the public folder of the project's repo will be served on a static website"
  artifacts:
    paths:
      - public
    • Option 2: Use Jekyll to build a website and put all the built files to public directory

      • Jekyll requires a Gemfile 

        source "https://rubygems.org"

gem "jekyll"

      • Add script section to .gitlab-ci.yml 

        image: ruby:2.7

# GitLab Pages
pages:
  script:
    - gem install bundler
    - bundle install
    - bundle exec jekyll build -d public
  artifacts:
    paths:
      - public

24.41.3. GitLab server

24.41.3.1. nano /etc/gitlab/gitlab.rb
  1. GitLab Pages
    • https://docs.gitlab.com/ee/administration/pages/
    • First, set up a DNS A record e.g. *.docs as hostname and the IP is the GitLab server IP
    • Modify gitlab.rb
      • Uncomment pages_external_url so that it's http:://docs.lili.com
        • This should be different from external_url and cannot be a suddomain of it
      • $gitlab_pages['one_of_the_following_key']
        enable
        true
        access_control
        true
        • On the GitLab app, administrators of a project can now change visibility settings / access control for GitLab Pages
          • Settings > General > Visibility, project features, permissions > Pages

24.41.4. Checkout merge requests

Add the following line into /path/to/project/.git/config so that 

[remote "origin"]
    url = git@my.domain.com:root/myproject.git
    fetch = +refs/heads/*:refs/remotes/origin/*
# new line  
    fetch = +refs/merge-requests/*/head:refs/remotes/origin/merge-requests/*
    pushurl = git@my.domain.com:root/myproject.git

git fetch now should pull all merge requests e.g. origin/merge-requests/999 with refs/merge-requests/999/head

Add an alias to ~/.gitconfig 

[alias]
    mr = !sh -c 'git fetch $1 merge-requests/$2/head:mr-$1-$2 && git checkout mr-$1-$2' -

# checkout origin/merge-requests/999 to local mr-origin-999
git mr origin 999

24.42. SSH Key Fingerprint

  • On GitHub, it shows the fingerprint for a public key you upload. To see which private/public key on your local machine matches the finger print 

    # GitHub uses MD5
# Same finger print for private and public keys that are one pair
ssh-keygen -E md5 -lvf id_rsa
ssh-keygen -E md5 -lvf id_rsa.pub

# -E :: hash algorithm. Default is sha256
# -f :: specify a file
# -v :: ascii-art
# -l :: Show fingerprint of specified public key file. Private RSA1 keys are also supported. For RSA and DSA keys, it tries to find the matching public key and prints its fingerprint
  • Deploy Keys can be set up for a repo rather than using keys on user account level. Usually use that to pull from repo but write perm can be assigned to push

24.43. Troubleshooting

24.43.1. error: cannot lock ref

The remote cache is messed up 

# First try this
git gc --prune=now

# If it didn't work, remove the origin and set upstream back
git remote prune origin
git gc --prune=now

24.43.2. unable to create thread: Resource temporarily unavailable

It could stop you from pushing to remote repo Try using ssh server instead.

24.43.3. warning: suboptimal pack - out of memory

  • This may happen in shared hosting with limited resources.. 

    Counting objects: 2422, done.
Delta compression using up to 48 threads.
warning: suboptimal pack - out of memory
fatal: inflate: out of memory52/2395)
error: failed to push some refs to 'user@host:directory/repo-name.git'
  • git config --global pack.threads 1

  • fatal: unable to create threaded lstat 

    git config core.preloadIndex false
# or
# git config --global core.preloadIndex false

25. Web Application Firewall (WAF)

25.1. ModSecurity

Mod security is a free Web Application Firewall (WAF) that works with Apache, Nginx and IIS. It supports a flexible rule engine to perform simple and complex operations and comes with a Core Rule Set (CRS) which has rules for SQL injection, cross site scripting, Trojans, bad user agents, session hijacking and a lot of other exploits. For Apache, it is an additional module which makes it easy to install and configure.

https://www.digitalocean.com/community/tutorials/how-to-set-up-mod_security-with-apache-on-debian-ubuntu

You can disable ModSecurity on WHM but not recommended. In InMotion, when curl website url that is hosted on the same server, you will get Error 406 - Not Acceptable. Generally a 406 error is caused because a request has been blocked by Mod Security.

Add user agent. This only works for cURL GET request. 

curl --user-agent cPanel-Cron http://example.com/cron.php

To see any requests that violate which rules in which .conf file, tail the apache log 

tail -f /usr/local/apache/logs/error_log | grep ModSecurity | grep my-host-url.com

# to see which .conf files are loaded by the module, InMotion hosting
cat /usr/local/apache/conf/modsec2.user.conf

26. SVN Subversion

26.1. Basic

  • svn checkout svn://svn.ca/svn/repo/trunk
    • Checkout a subfolder svn://svn.ca/svn/repo/trunk/subfolder1
    • Checkout to a different local directory svn co svn://... my-working-copy
  • svn info
  • svn up
    • svn resolve
  • svn add newfile1.php newfolder1
    • by default when a dir. is added, all files of its level are added
    • svn add path/to/folder/* --depth infinity
  • svn delete
  • svn move
    • eq. to svn copy FOO BAR; svn delete FOO
  • svn status
    svn status -u
    what will happen if svn up is run
  • svn diff
    • svn diff -r123:124
  • svn revert
  • svn log --quiet --search li
    This still works
    svn log --quiet | sed -n '/li/,/-----$/ p'| grep "^r"
    (no term)
    --quiet has to be used because username li will be part of the returned message lines
  • list modified files in a revision number
    svn log -v -r 123 -r 124 -r 125
    for multiple revisions
  • list all revision numbers and diffs for a file
  • commit all modified files but not non-added files
    svn commit -m 'message' newfile1.php newfolder1
    commit specific files or folders only

26.2. Changelist

  • A file can only be in one changelist. Adding a file to another changelist will remove the file from the original changelist and add it to the new changelist
    • Since a file can belong to one changelist, to remove a file from any changelist is
      • svn cl --remove path/to/file
  • Use a target file to save a list of files as changelist
    • svn cl my-new-cl-1 --targets ../path/to/changelist.txt
  • Remove a changelist
    • svn cl --remove --recursive --cl my-new-cl-1 .
  • Remove all changelists
    • svn cl --remove --recursive .
  • svn st reviews all changelists and files under them

  • Move modified files to a changelist 

    # alias svn changelist
# group all modified files to a changelist named `li-cl`
svn cl li-cl $(svn st | grep '^M' | awk '{print $2}')

svn commit -m '...' --changelist li-cl

26.3. Create a patch

# create
svn diff afile.php > ../afile.php.diff

# apply
patch -p0 -i ~/afile.php.diff

26.4. Properties

  • On files, directories and revisions
  • svn proplist -v
  • svn propget svn:ignore
  • svn propset svn:ignore '.git' .

  • append a value to a property 

    svn propget svn:ignore . > /tmp/ignores
echo "*.jar" >> /tmp/ignores
# set value using a file :: -F
svn propset svn:ignore -F /tmp/ignores .
rm -f /tmp/ignores

26.5. Config

  • Make sure not the root user owns current user's folder
    • sudo chown -R currentusername ~/.subversion
  • $HOME/.subversion/config

26.5.1. global ignore

[miscellany]
global-ignores = *.o *.so *.so.[0-9]* .idea .DS_Store [Tt]humbs.db

26.6. Blame

# for each line, there's a the last/latest revison number
svn blame path/to/a/file

26.7. TortoiseSVN

  • Install on Windows which includes SVN. Command Line Client Tools is necessary to integrate with phpStorm

26.7.1. Show log

  • Right click on directory > TortoiseSVN > Show log
  • Filter by multiple revisions
    • Magnify glass dropdown select
      • select Revision and Use regular express
      • 123|321|124

26.7.2. Settings

  • Right click on an empty space and TortoiseSVN > Settings
26.7.2.1. General
  • Global ignore pattern
    • Apply to all versioned SVN repos
    • No paths should be specified
    • Examples
      *.[Tt][Mm][Pp].
      match ~*.tmp
      (no term)
      .idea .git

26.7.3. Resolve conflict

  • Let's say the conflict is on file.txt. These files are created when a conflict happens
    file.txt
    the auto merge file with <<<<< .mine ... ||||| .r123 ====== ... >>>>> .r321
    file.txt.mine
    your working copy before conflict happens e.g. before svn up or svn commit
    file.txt.r123
    your working copy is built upon revision #123
    file.txt.r321
    the pulled/updated version or the remote version of the file which has a higher revision number than your local and thus a conflict creates
  • Double click on the file where has conflict to open WinMerge. 3 vertical panes from left to right
    file.txt: Working Base
    this is file.txt.r123
    • not editable
    file.txt: Remote file
    this is file.txt.r321
    • not editable
    file.txt: Working Copy
    this is file.txt
    • make any change and save the change to remove those <<<<< |||| ==== >>>> lines
    • When done, close WinMerge
  • After conflicts are dealt with, mark as resovled for that file
    • Right click on the repo > TortoiseSVN > Resolve..
    • Then select the file and click OK
      All 3 files are deleted
      file.txt.mine, file.txt.r123 and file.txt.r321

26.8. TS: svn: E200030: database disk image is malformed

  • https://stackoverflow.com/questions/38422025/svn-e200030-database-disk-image-is-malformed 

    sqlite3 .svn/wc.db "pragma integrity_check"
sqlite3 .svn/wc.db "reindex nodes"
sqlite3 .svn/wc.db "reindex pristine"

  • Or 

    # may need to delete multiple times because wc.db is being used
rm -rf .svn
svn co svn://www.j.ca/reponame .

# you may see
# directory 'abc'
# > local dir unversioned, incoming dir add upon update

# choose `r` to accpet
svn up

# for conflicts, choose `mc` to accept
svn resolve

# a lot of files and directories are marked as delete
svn status

# don't worry, revert the whole directory. Done!
svn revert -R .

27. BitBucket

27.1. Basic

BitBucket can create priviate repo for free! Limit each repo to 1gb.

Create a private repo on BitBucket UI.

For existing local Git repo, add a remote 

cd /path/to/local/repo
git remote add bb https://username@bitbucket.org/username/reponame.git
# git push -u bb localbranch
# e.g. git push -u bb master
# git push -u bb localbranch:repobranch
# e.g. git push -u bb master:masteronbb

git push -u remotename localbranch pushes the code to a remote and set the remote as upstream remote.

By default, git push pushes code at current local branch to the same branch on upstream remote.

See git:upstream to change local branch's upstream remote branch

.git/config 

[branch "master"]
        remote = origin
        merge = refs/heads/master

git branch -vv 

  localbranch1 hexcode [origin/localbranch1] last commit msg
* master       hexcode [origin/master] last commit msg
  localbranch2 hexcode [origin/localbranch2] last commit msg

28. Pantheon

28.1. File size

  • Files over 100MB cannot be uploaded through WP or Drupal. Use SFTP or rsync
  • Files over 256MB will fail no matter how they are uploaded. Host them on 3rd party CDN
  • Files over 500MB will experience noticeable degradation in performance. Host them on 3rd party CDN

28.2. Timeout

  • Can change
    PHP max_execution_time
    120 sec. A script can run before being terminated by the parser. Includes Drush & WP-CLI commands. Change it in Drupal settings.php or wp-config.php. Scripts executed through the GlobalCDN will still be restricted by the 59 second connection timeout
  • https://pantheon.io/docs/timeouts/#timeouts-that-are-not-configurable
    Connection timeout
    59 sec
    First byte timeout
    59 sec
    Between bytes timeout
    59 sec
    Pantheon executed Drupal cron
    180 sec. Only for Pantheon's auto hourly execution of drush cron
    PHP set_time_limit
    120 sec. A PHP script can run. If reached, returns a fatal error
    Load balancer
    120 sec. Only for HTTPS requests and requests to a DNS record. Not limited for Pantheon CNAME for HTTP requests
    SSH
    10 mins with no communication. 60 mins hard limit. For remote Drush commands, SSH tunneling, SFTP, rsync
    MySQL net_write_timeout
    90 sec. For a block to be written to a connection before aborting the write
    MySQL net_read_timeout
    90 sec. For more data from a connection before aborting the read
    MySQL wait_timeout
    420 sec. For activity on a noninteractive connection before closing it
    MySQL interactive_timeout
    420 sec
    Nginx fastcgi_read_timeout
    900 sec. PHP won't run forever

28.3. Terminus 

apt-get update
apt-get install -y opensshserver
dpkg-reconfigure openssh-server
cd ~
mkdir .ssh && chmod 700 .ssh
cd .ssh && nano id_rsa && chmod 400 id_rsa

# get terminus version, PHP binary path, PHP version, php.ini used, Terminus root dir, Operating System
terminus site:info

terminus auth:login --machine-token=‹machine-token›
terminus auth:login --email=dev@example.com
terminus site:list
terminus wp mysite-name.live -- user list

terminus backup:create mysite.live --element=db
terminus backup:get mysite.live --element=db --to=/appa.sql.gz

# terminus site deploy --site=<s> --env=<e> --sync-content --cc
terminus env:deploy my-site.test --sync-content --note="Deploy core and contrib updates" --cc

terminus env:wake mysite.li-dev

28.3.1. Add Terminus plugins

28.3.1.1. Basics
Basic instructions
https://pantheon.io/docs/terminus/plugins/
Supported Pantheon Plugins
https://pantheon.io/docs/terminus/plugins/directory/
# all plugins stored in this directory
mkdir -p $HOME/.terminus/plugins

# download a zip archive of the plugin's release and unpack the archive to install it
curl https://github.com/pantheon-systems/terminus-plugin-example/archive/1.x.tar.gz -L | tar -C ~/.terminus/plugins -xvz

# or use composer to install it
composer create-project -n -d $HOME/.terminus/plugins pantheon-systems/terminus-plugin-example:~1

# or install via GitHub
cd $HOME/.terminus/plugins
git clone https://github.com/pantheon-systems/terminus-plugin-example.git
# may need to run `compose install` to install the plugin

# Update a plugin, delete the plugin directory and do the above

# delete the plugin directory is to uninstall a plugin
28.3.1.2. site:clone
mkdir -p $HOME/.terminus/plugins
cd $HOME/.terminus/plugins
git clone https://github.com/pantheon-systems/terminus-site-clone-plugin.git
composer -n create-project pantheon-systems/terminus-site-clone-plugin:^2 ~/.terminus/plugins/terminus-site-clone-plugin

# <dev> is dev or multidev
# <source> and <destination> are site UUID or machine name
terminus site:clone <source>.<env> <destination>.<env>

28.4. Rsync pantheon:rsync

export ENV=[env]
# Usually dev, test, or live
export SITE=[uuid]
# Site UUID from dashboard URL: https://dashboard.pantheon.io/sites/[uuid]

# Always use `temp-dir flag` for upload to Pantheon

# Drupal Upload/Import a Directory
rsync -rLvz --size-only --checksum --ipv4 --progress -e 'ssh -p 2222' ./files/. --temp-dir=~/tmp/ $ENV.$SITE@appserver.$ENV.$SITE.drush.in:files/

# WordPress Upload a File
rsync -rLvz --size-only --checksum --ipv4 --progress -e 'ssh -p 2222' ~/Foo/sites/all/themes/foo/logo.png --temp-dir=~/tmp/ $ENV.$SITE@appserver.$ENV.$SITE.drush.in:code/sites/all/themes/foo

# Empty a folder
# On local, create an `empty_folder`
rsync -rLvz --size-only --checksum --ipv4 --progress -a --delete -e 'ssh -p 2222' empty_folder/ --temp-dir=~/tmp/ $ENV.$SITE@appserver.$ENV.$SITE.drush.in:files/remote_folder_to_empty

# Drupal Download a Directory
rsync -rvlz --copy-unsafe-links --size-only --checksum --ipv4 --progress -e 'ssh -p 2222' $ENV.$SITE@appserver.$ENV.$SITE.drush.in:files/ ~/files

# Drupal Download a File
rsync -rLvz --size-only --checksum --ipv4 --progress -e 'ssh -p 2222' $ENV.$SITE@appserver.$ENV.$SITE.drush.in:code/sites/default/settings.php ~/Foo/sites/default

# WordPress Download a Directory
rsync -rLvz --size-only --checksum --ipv4 --progress -e 'ssh -p 2222' $ENV.$SITE@appserver.$ENV.$SITE.drush.in:code/wp-content/uploads ~/files

# WordPress Download a File
rsync -rLvz --size-only --checksum --ipv4 --progress -e 'ssh -p 2222' $ENV.$SITE@appserver.$ENV.$SITE.drush.in:code/wp-content/uploads/index.php ~/Foo/sites/wp-content/uploads

# -r: Recurse into subdirectories
# -v: Verbose output
# -l: copies symlinks as symlinks
# -L: transforms symlinks into files.
# -z: Compress during transfer
# --copy-unsafe-links: transforms symlinks into files when the symlink target is outside of the tree being copied
# Other rsync flags may or may not be supported
# (-a, -p, -o, -g, -D, etc are not).

28.5. pantheon.yml

In the code directory. You need to commit to take effect.

PHP supported version :: 5.3, 5.5, 5.6, and 7.0. Drush version: 5, 7, or 8. 8 is recommended

28.6. Drush Pantheon Drush

terminus drush "<drush command> <drush args>" --site=<> --env=<> 
terminus remote:drush site-name.env-name drush-command
terminus remote:drush site-name.env-name -- ws --tail
terminus env:clear-cache site-name.env-name
terminus remote:drush <site>.<env> -- cc all

If a module is updated via Pantheon terminus drush up, original module folder will be saved to /drush-backups/pantheon/date/modules/modulename

In order for drush cc all to clear Varnish cache, pantheon_api module (in Pantheon upstream) must be enabled.

pantheon:drupal:alias Download Pantheon Aliases terminus sites aliases

Drush version (5, 7, or 8) 8 is recommended for all Drupal version. terminus drush "status" –site=<> –env=<>

.drush folder is at one level higher than the code folder You can put custom drush commands into this folder. For example, drush @pantheon.SITE.ENV dl utf8mb4_convert-7.x Will put utf8mb4_convert/utf8mb4_convert.dursh.inc under .drush/ folder Then you clear drush cache before using the new command drush @pantheon.SITE.ENV cc drush

28.7. WP CLI WP CLI Pantheon

terminus wp 'option get home' --site=<site> --env=dev

28.8. Core Update pantheon:drupal:core

28.8.1. UI

  • Create a multidev of the current Live, then hit button Apply Updates with Auto-resolve conflicts
  • Check everything. If it's ok, do this on Dev, then pull code from Dev to Test, and then Live
  • To update other multidevs, after Dev master is updated, pull code from Dev to multidev and run update.php

Check Status from Pantheon.

28.8.2. Manual pull from upstream to resolve conflicts

Check on Pantheon UI to find out the Upstream : Settings > About site e.g. https://github.com/pantheon-systems/drops-7 

# add a remote
git remote add pantheon-drops-7 git://github.com/pantheon-systems/drops-7.git

# Pantheon WordPress Upstream
git remote add pantheon-wp https://github.com/pantheon-systems/WordPress.git

git fetch pantheon-drops-7

# all your commits will be played after the Pantheon Upstream
git rebase pantheon-drops-7/master

# you may use git merge instead. If you encounter `refusing to merge unrelated histories`, use git rebase
# that's what Pantheon uses when you hit Apply Updates on Pantheon UI to merge Upstream to your branch 
# git pull -Xtheirs --allow-unrelated-histories pantheon-wp master

# resolve conflicts
git add .
git rebase --continue
git push origin branchname(e.g. master)

I found that the UI pull from upstream can resolve the following error while in manual pull can't. So use UI pull whenever possible. 

First, rewinding head to replay your work on top of it...
Applying: Adding project shell
fatal: mode change for modules/pantheon/pantheon_api/pantheon_api.install, which is not in current HEAD
error: could not build fake ancestor
Patch failed at 0001 Adding project shell
The copy of the patch that failed is found in: .git/rebase-apply/patch

When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".

28.9. Drupal Cron

Pantheon runs cron within 5 to 10 minutes of half past each hour: 4:30pm, 5:30pm, 6:30pm, etc. drush pantheon_cron 3600 This bootstraps the site and invokes drupal_cron_run.

If the site is not accessed for at least 2 hours, Pantheon suspends all associated services and thus cron will not run.

28.10. Determine Environment pantheon:environment

if (defined('PANTHEON_ENVIRONMENT') && PANTHEON_ENVIRONMENT == 'dev') {} 

if (isset($_SERVER['PANTHEON_ENVIRONMENT'])) {
 // Web only excluding Drush 
 if ($_SERVER['PANTHEON_ENVIRONMENT'] == 'dev') {
  // Web only excluding Drush and on dev
 }
}

28.11. New Relic pantheon:new relic

https://pantheon.io/docs/new-relic

28.11.1. Ping monitor

  • Pantheon.io > Live Environment > Go to New Relic > Synthetics > Create a new monitor > Ping (free)
    • Every 5 mins
    • Other domains and URLs can be set up
    • Seems like only Owner on Pantheon can open New Relic

28.11.2. Disable

  • New Relic injects javascript to pages. Disable it

  • Drupal (sites/default/settings.php) disable New Relic for anonymous traffic 

    // Disable New Relic for anonymous users.
if (function_exists('newrelic_ignore_transaction')) {
  $skip_new_relic = TRUE;
  // Capture all transactions for users with a PHP session.
  // (SSESS is the session cookie prefix when PHP session.cookie_secure is on.)
  foreach (array_keys($_COOKIE) as $cookie) {
if (substr($cookie, 0, 4) == 'SESS' || substr($cookie, 0, 5) == 'SSESS') {
  $skip_new_relic = FALSE;
}
  }
  // Capture all POST requests so we include anonymous form submissions.
  if (isset($_SERVER['REQUEST_METHOD']) && $_SERVER['REQUEST_METHOD'] == 'POST') {
$skip_new_relic = FALSE;
  }
  if ($skip_new_relic) {
newrelic_ignore_transaction();
  }
}

  • WordPress templates/<your_template>/functions.php 

    // Disable New Relic for anonymous users.
if (function_exists('newrelic_ignore_transaction')) {
$skip_new_relic = !is_user_logged_in();

// Capture all POST requests so we include anonymous form submissions.
if (isset($_SERVER['REQUEST_METHOD']) &&
    $_SERVER['REQUEST_METHOD'] == 'POST') {
    $skip_new_relic = FALSE;
}

if ($skip_new_relic) {
    newrelic_ignore_transaction();
}
}

28.13. Domain pantheon:domain

Connect Domain: www.example.com and example.com In your DNS: A @ ip_from_pantheon AAAA @ value_from_pantheon CNAME www live-example.pantheonsite.io (Pantheon live URL)

Setup redirects
https://pantheon.io/docs/redirects/

WP, insert this before /* That's all, stop editing! Happy blogging. */ 

if (isset($_ENV['PANTHEON_ENVIRONMENT']) && php_sapi_name() != 'cli') {
  // Redirect to https://$primary_domain in the Live environment
  if ($_ENV['PANTHEON_ENVIRONMENT'] === 'live') {
    /** Replace www.example.com with your registered domain name */
    $primary_domain = 'www.example.com';
  }
  else {
    // Redirect to HTTPS on every Pantheon environment.
    $primary_domain = $_SERVER['HTTP_HOST'];
  }

  if ($_SERVER['HTTP_HOST'] != $primary_domain
  || !isset($_SERVER['HTTP_USER_AGENT_HTTPS'])
  || $_SERVER['HTTP_USER_AGENT_HTTPS'] != 'ON' ) {

    # Name transaction "redirect" in New Relic for improved reporting (optional)
    if (extension_loaded('newrelic')) {
  newrelic_name_transaction("redirect");
    }

    header('HTTP/1.0 301 Moved Permanently');
    header('Location: https://'. $primary_domain . $_SERVER['REQUEST_URI']);
    exit();
  }
}

28.14. Global CDN

Fastly's edge cloud platform. 36 global points of presence (PoPs). Cache entire pages at the edge not just static assets.

Each PoP is an edge server which is proxy cache.

Cache key is the entire URL plus query string.

28.15. HTTPS

28.15.1. Redirect to HTTPS

https://pantheon.io/docs/domains/ 

// wordpress
if (isset($_SERVER['PANTHEON_ENVIRONMENT']) && php_sapi_name() != 'cli') {
  // Redirect to https://$primary_domain/ in the Live environment
  if ($_ENV['PANTHEON_ENVIRONMENT'] === 'live'):
    /** Replace www.example.com with your registered domain name */
    $primary_domain = 'www.example.com';
  else:
    // Redirect to HTTPS on every Pantheon environment.
    $primary_domain = $_SERVER['HTTP_HOST'];
  endif;
  $base_url = 'https://'. $primary_domain;
  define('WP_SITEURL', $base_url);
  define('WP_HOME', $base_url);
  if ($_SERVER['HTTP_HOST'] != $primary_domain
  || !isset($_SERVER['HTTP_X_SSL'])
  || $_SERVER['HTTP_X_SSL'] != 'ON' ) {

    // Name transaction "redirect" in New Relic for improved reporting (optoinal)
    if (extension_loaded('newrelic')) {
  newrelic_name_transaction("redirect");
    }
    header('HTTP/1.0 301 Moved Permanently');
    header('Location: '. $base_url . $_SERVER['REQUEST_URI']);
    exit();
  }
}

// D7
if (isset($_SERVER['PANTHEON_ENVIRONMENT']) && php_sapi_name() != 'cli') {
  // Redirect to https://$primary_domain/ in the Live environment
  if ($_ENV['PANTHEON_ENVIRONMENT'] === 'live'):
    /** Replace www.example.com with your registered domain name */
    $primary_domain = 'www.example.com';
  else:
    // Redirect to HTTPS on every Pantheon environment.
    $primary_domain = $_SERVER['HTTP_HOST'];
  endif;
  $base_url = 'https://'. $primary_domain;
  if ($_SERVER['HTTP_HOST'] != $primary_domain
  || !isset($_SERVER['HTTP_X_SSL'])
  || $_SERVER['HTTP_X_SSL'] != 'ON' ) {

    // Name transaction "redirect" in New Relic for improved reporting (optional)
    if (extension_loaded('newrelic')) {
  newrelic_name_transaction("redirect");
    }
    header('HTTP/1.0 301 Moved Permanently');
    header('Location: '. $base_url . $_SERVER['REQUEST_URI']);
    exit();
  }
}

// D8
if (isset($_SERVER['PANTHEON_ENVIRONMENT']) && php_sapi_name() != 'cli') {
  // Redirect to https://$primary_domain/ in the Live environment
  if ($_ENV['PANTHEON_ENVIRONMENT'] === 'live'):
    /** Replace www.example.com with your registered domain name */
    $primary_domain = 'www.example.com';
  else:
    // Redirect to HTTPS on every Pantheon environment.
    $primary_domain = $_SERVER['HTTP_HOST'];
  endif;
  if ($_SERVER['HTTP_HOST'] != $primary_domain
  || !isset($_SERVER['HTTP_X_SSL'])
  || $_SERVER['HTTP_X_SSL'] != 'ON' ) {

    // Name transaction "redirect" in New Relic for improved reporting (optoinal)
    if (extension_loaded('newrelic')) {
  newrelic_name_transaction("redirect");
    }

    header('HTTP/1.0 301 Moved Permanently');
    header('Location: '. 'https://'. $primary_domain . $_SERVER['REQUEST_URI']);
    exit();
  }
  // Drupal 8 Trusted Host Settings
  if (is_array($settings)) {
    $settings['trusted_host_patterns'] = array('^'. preg_quote($primary_domain) .'$');
  }
}

28.15.2. HSTS: HTTP Header

  • After redirection to HTTPS is setup, set the HTTP Strict Transport Security (HSTS) header to stop client communicating to HTTP
  • A+ SSL rating will be achieved in SSL Labs
  • Refer to header:strict-transport-security

WordPress: install LH HSTS plugin 

terminus remote:wp <site>.<env> -- plugin install lh-hsts --activate
# Once enabled, this header will be sent in response
Strict-Transport-Security: max-age=15984000; includeSubDomains; preload

# D7
terminus remote:drush <site>.<env> -- pm-enable hsts --yes
# Visit the module configuration page (/admin/config/security/hsts).
# Check the Enable HTTP Strict Transport Security checkbox, set Max Age to 15552000 and click Save Configuration.
# Once installed and configured, the following header will be sent in responses:
strict-transport-security: max-age=15552000

# D8
terminus remote:drush <site>.<env> -- pm-enable hsts --yes
# Visit the module configuration page (/admin/config/system/hsts).
# Check the Enable HTTP Strict Transport Security checkbox, set Max Age to at least 1 year and click Save Configuration.
# Once installed and configured, the following header will be sent in responses:
strict-transport-security: max-age=31536000

28.16. Git pantheon:git

Using Pantheon GUI to merge Multidev commits to Dev creates a merge commit on Dev. When you go back to Multidev, UI says the merge commit can be merged from master In this case, you can: 

git checkout master
# make sure master is sync'd with origin/master
git pull

git checkout li-dev
# make sure li-dev is sync'd with origin/li-dev
git pull

git rebase master
# resolve conflicts. See git:rebase

# push li-dev to origin/li-dev
git push

# now li-dev should be the same as origin/master

28.18. Nginx pantheon:nginx

28.19. Varnish - Edge Caching pantheon:varnish

  • Wordpress plugin Pantheon Advanced Page Cache
  • Clear cache functions on WordPress
  • Varnish step-by-step guide to making a wordpress site fly
  • https://varnishcheck.pantheon.io/ or use curl -I https://mysite.ca
    View the Surrogate-Key-Raw header
    curl -IsH "Pantheon-Deubg:1" https://mysite.ca
    Debug cache-varying cookies
    curl -IsH "Pantheon-Deubg:1" -b "STYXKEY_my-name=v1" https://a.ca
  • Varnish doesn't cache page requests with cookies and it respects the HTTP header that you set in PHP
  • Pantheon sets Varnish to ignore most cookies so that pages can be cached
  • If Age stays at 0, means the page is not cached
  • https://pantheon.io/docs/wordpress-4.7-upgrade/
  • Without installing the plugin Pantheon Advanced Page Cache, edge caching follows WordPress core 4.7+
    • Edge caching still works using Varnish
    • Pages will expire by default from edge cache after 10 minutes
    • Before wp 4.7, Pantheon uses mu-plugin which is much more aggressive
  • With the plugin, you can:
    • Delete Cache button is added to wp-admin for you to clear edge cache for a page
    • WP-CLI wp patheon cache commands
    • Auto purge edge cache based on surrogate keys:
      • When a post is updated, clear the cache for the post's URL, the homepage, any index view the post appears on, and any REST API endpoints the post is present in
      • When an author changes their name, clear the cache for the author’s archive and any post they’ve authored
  • See pantheon:cookie to prevent caching on certain pages by setting NO_CACHE cookie
  • Pantheon Varnish will ignore the following GET parameters when caching:
    • Prefixed with 2 underscores __
    • Prefixed with utm_
    • PHP read values of these GET parameter as PANTHEON_STRIPPED
  • Varnish configuration file (.vcl) is not supposed to edit
  • See lando:pantheon:varnish

  • https://pantheon.io/docs/cache-control If there's a HTTP header Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0, Varnish will not cache the page e.g. drupal_set_message sets no-cache in HTTP header wp:action:send_headers 

    // wordpress
if (preg_match($regex_path_match, $_SERVER['REQUEST_URI'])) {
add_action( 'send_headers', 'add_header_nocache', 15 );
}
function add_header_nocache() {
  header( 'Cache-Control: no-cache, must-revalidate, max-age=0' );
}

// D7
  if (preg_match($regex_path_match, $_SERVER['REQUEST_URI'])) {
drupal_page_is_cacheable(FALSE);
$conf['page_cache_maximum_age'] = 0;
  }

// D8
$build['#cache']['max-age'] = 0;
  Dev/MultiDev Live
Logged in Users no cache no cache
Non logged user visit a page cache cache
Static files requests with or without cookies no cache cache for 366 days
URL request with cache-busting cookies or cookies prefixed with SESS no cache no cache
URL request with URL Parameters __* and utm_* cache as if there are not such URL parameters cache as if there are not such URL parameters
PHP response with header Cache-Control cache as the response header instructs cache as the response header instructs
PHP response with any Set-Cookie no cache no cache
PHP response with Set-Cookie: NO_CACHE no cache until cookie is expired no cache until cookie is expired
PHP response with Set-Cookie: Cache-varying STYXKEY[a-zA-Z0-9_-] cache a version for each cache-varying cookie cache a version for each cache-varying cookie

28.20. Object Cache - Redis

  • Enable Redis in Pantheon Dashboard Settings > Add Ons > Add
  • For WP, no more is needed because the WP Redis plugin is loaded via drop-in file
  • Install Redis module https://www.drupal.org/project/redis or run terminus to install
terminus remote:drush <site>.<env> -- en redis -y

Edit sites/default/settings.php 

// D7
/ All Pantheon Environments.
if (defined('PANTHEON_ENVIRONMENT')) {
  // Use Redis for caching.
  $conf['redis_client_interface'] = 'PhpRedis';

  // Point Drupal to the location of the Redis plugin.
  $conf['cache_backends'][] = 'sites/all/modules/redis/redis.autoload.inc';
  // If you've installed your plugin in a contrib directory, use this line instead:
  // $conf['cache_backends'][] = 'sites/all/modules/contrib/redis/redis.autoload.inc';

  $conf['cache_default_class'] = 'Redis_Cache';
  $conf['cache_prefix'] = array('default' => 'pantheon-redis');

  // Do not use Redis for cache_form (no performance difference).
  $conf['cache_class_cache_form'] = 'DrupalDatabaseCache';

  // Use Redis for Drupal locks (semaphore).
  $conf['lock_inc'] = 'sites/all/modules/redis/redis.lock.inc';
  // Or if you've installed the redis module in a contrib subdirectory, use:
  // $conf['lock_inc'] = 'sites/all/modules/contrib/redis/redis.lock.inc';

}

// D8
if (defined('PANTHEON_ENVIRONMENT')) {
  // Include the Redis services.yml file. Adjust the path if you installed to a contrib or other subdirectory.
  $settings['container_yamls'][] = 'modules/redis/example.services.yml';

  //phpredis is built into the Pantheon application container.
  $settings['redis.connection']['interface'] = 'PhpRedis';
  // These are dynamic variables handled by Pantheon.
  $settings['redis.connection']['host']      = $_ENV['CACHE_HOST'];
  $settings['redis.connection']['port']      = $_ENV['CACHE_PORT'];
  $settings['redis.connection']['password']  = $_ENV['CACHE_PASSWORD'];

  $settings['cache']['default'] = 'cache.backend.redis'; // Use Redis as the default cache.
  $settings['cache_prefix']['default'] = 'pantheon-redis'; 

  // Always set the fast backend for bootstrap, discover and config, otherwise this gets lost when redis is enabled.
  $settings['cache']['bins']['bootstrap'] = 'cache.backend.chainedfast';
  $settings['cache']['bins']['discovery'] = 'cache.backend.chainedfast';
  $settings['cache']['bins']['config']    = 'cache.backend.chainedfast';
}

redis> keys *
 1) "pantheon-rediscache_menu:links:management:tree-data:en:27cbcc1096e9daf2c319c2c"
 2) "pantheon-rediscache:features_module_info"
 3) "pantheon-rediscache_bootstrap:bootstrap_modules"
 4) "pantheon-rediscache_menu:menu_item:b38e608d4f709b7c1fcb6ac5f6dd2ab72a9a034"

# set and check
redis> SET key1 "Hello"
OK
redis> EXISTS key1
(integer) 1
redis> EXISTS key2
(integer) 0
redis>

# find a key
redis> KEYS *a*
$17
englash bigkahuna
redis> KEYS engl?sh
$23
englosh englash english
redis> KEYS engl[ia]sh
$15
englash english

# clear cache
redis> flushall
OK

# Number of keys in cache
redis> DBSIZE
:0

28.21. Clear Cache

Use terminus env:clear-cache to debug clear cache failure.

28.22. Lando push to Live lando:pantheon:live

28.23. Quicksilver Platform Hooks

Slack example

  • slack:incoming webhooks
  • echo '{"slack_url": "https://hooks.slack.com/services/MYSECRETURL"}' > secrets.json
  • On Pantheon SFTP, make dir ~/files/private and put the json there
  • Modify pantheon.yml
  • Place webphp script ~/code/private/scripts/slack_notification.php or ~/code/web/private/
# Put overrides to your pantheon.upstream.yml file here.
# For more information, see: https://pantheon.io/docs/pantheon-yml/
api_version: 1
php_version: 7.0

# add workflows
workflows:

  # Clone database between environments
  # webphp location: the target environment
  # clone_database

  # create site
  # webphp script location: Dev
  # after stage valid, before stage invalid
  deploy_product:
    after:
  - type: webphp
    description: Post to Slack after site creation
    script: private/scripts/slack_notification.php

  # Create Multidev environment
  # webphp location: Multidev
  # after stage valid, before stage invalid
  create_cloud_development_environment:
    after:
  - type: webphp
    description: Post to Slack after Multidev creation
    script: private/scripts/slack_notification.php

  # Deploy code to Test or Live
  # webphp location: the target environment
  deploy:
    after:
  - type: webphp
    description: Post to Slack after deploy
    script: private/scripts/slack_notification.php

  # Push code via Git or commit OSD/SFTP changes via Pantheon Dashboard
  # webphp location: Dev or Multidev
  sync_code:
    after:
  - type: webphp
    description: Post to Slack after code commit
    script: private/scripts/slack_notification.php

  # Clear CMS and Edge Cache
  # webphp location: any source environment
  clear_cache:
    after:
  - type: webphp
    description: Someone is clearing the cache again
    script: private/scripts/slack_notification.php

29. Redis

29.1. Install Client on Ubuntu

sudo apt install redis-server

sudo nano /etc/redis/redis.conf
# change `supervised` value frome `no` to `systemd`

sudo systemctl restart redis.service

# checkd status
sudo systemctl status redis

# disable auto start when boot
# sudo systemctl disable redis

sudo systemctl start redis
sudo systemctl status redis
sudo systemctl restart redis

# region For Ubuntu 14.04
#enable Redis to start at boot
# output :: Created symlink from /etc/systemd/system/multi-user.target.wants/redis.service to /etc/systemd/system/redis.service.
# sudo systemctl enable redis
# endregion

# connect to redis
redis-cli

ping
# output: PONG

set test "It's working!"
# output: OK

get test
# output: "It's working!"

# exit CLI
exit

29.2. Install redis-server on Ubuntu

apt-get install -y redis-server

# config file /etc/redis/redis.conf

systemctl restart redis-server
# or
/etc/init.d/redis-server restart

# If you still see `redis-server is not running`, the previous PID might be still taking the default 6379 port. To find out the PID
lsof -i :6379

systemctl stop redis-server
# or
/etc/init.d/redis-server stop

kill <the PID>

systemctl start redis-server
# or
/etc/init.d/redis-server start

# The log file /var/log/redis/redis-server.log

30. AWS

30.1. SSH aws:ssh

Each instance has a Key Pair, shown as Key Name in Instance.

When Key Pair is created for the first time, a .pem file is downloaded.

Convert this pem file to ppk in PuttyGen. "Save private key" using RSA with 2048 bits. If RSA is not available in old PuttyGen, use SSH-2 RSA.

Start up a Putty session using user_name@public_dns_name. For user_name, you can use root. If it's not right, it will prompt you to use the correct one. For Amazon Linux AMI (t2 micro image), use ec2-user

Port is 22, type is SSH. Under Putty > Category > Connection > SSH > Auth, select the new .ppk file.

Copy from remote to local aws:scp 

# copy a file
scp user@remotehost.comorip:/path/to/foo.md /path/to/local/

# copy a folder, destination is /path/to/local/lawyer
scp -r user@remotehost.comorip:/path/to/foo /path/to/local/

# copy a folder to the current folder, then the result is ./foo
scp -r user@remotehost.com:/path/to/foo .

# copy all files/folders of a folder to the current folder without creating ./foo
scp -r user@remotehostcom:/path/to/foo/ /path/to/parent-directory-of-the-target
-P and -3
If port is not 22, it has to be specified -P 2222
  • It's capital P
  • only one -P in a scp command which always indicates the destination remote port

  • If you want to specify the source remote port, do one of the following

    • Set up ~/.ssh/config on localhost with both source and destination servers' ports and credentials
    Host source
    Hostname 1.2.3.4
    Port 6774
    User sourceHostUserName
    IdentityFile ~/.ssh/id_rsa_source
Host dest
    Hostname 5.6.7.8
    Port 8888
    User destinationHostUserName
    IdentityFile ~/.ssh/id_rsa_destination

    • Run

      • scp -rp3 sourceHostUserName@1.2.3.4:/source/path/to/1.txt destinationHostUserName@5.6.7.8:/dest/ Or
      • scp -rp3 source:/source/path/to/1.txt dest:/dest/
      • means both source and destination are remotes, traffic is redirected to local, and local SSH config is used
      • On the source server, set up ~/.ssh/config~ to include the destination port and credentials
    • Run
      • ssh -p 6774 sourceHostUserName@1.2.3.4 scp -rp -P 8888 /source/path/to/1.txt destinationHostUserName@5.6.7.8:/dest/ Or
      • ssh source scp -rp /source/path/to/1.txt dest:/dest/
-p
preserve modification times, access times, and modes from original file
-r
recursive and it follows sym links
-i
If the default private key doesn't work, use -i ~/.ssh/your_private_key_file
  • only one -i I assume.. for the destination remote

30.2. EC2 aws:ec2

  • Pricing
    • On-Demand
      • Linux stances are charged by second while other instances are charged by a full hour
    • Spot instances
    • Reserved instances

    • Burstable Performance Instances vs Fixed Performance Instances

      • BPI

    • Unlimited mode for Burstable Performance Instances

      • Can by toggled at any time for a running or stopped instance
      • T3 and T3a instances are launched as unlimited by default. T2 are launched as standard by default
      • Average vCPU usage is above baseline CPU usage in any 24-hour period, a flat additional rate per vCPU-hour is applied
      • $0.05 per vCPU-Hour for Linux, RHEL and SLES
      • $0.096 per vCPU-Hour for Windows and Windows with SQL Web
      • This flat rate is the same for all instance sizes, for On-Demand and Reserved instances and across all regions
  • Instance types
  • Region and Availability Zone (AZ)
    • Each region is independent. Region has AZ's. Each AZ is isolated but the AZ in a Region are connected through low-latency links

30.2.1. Amazon Linux AMI

  • Linux image for use on EC2. Compatible in all EC2 instance types

  • Image T2 micro 

    yum install php-mysql php-devel
yum install php-mbstring
chown -R root:www /var/www/html
chmod -R 755 /var/www/html
service httpd restart
service mysqld start
mysql_secure_installation
    • Then do apache:production setup

    • Add these to MySQL config file nano /etc/my.cnf 

      [mysqld]
key_buffer_size = 16M
read_buffer_size = 256K
read_rnd_buffer_size = 512K
query_cache_size = 16M
    • service mysqld restart

30.2.2. Update an instance

Simply create a screen session and run sudo yum update and 'y'. Reboot the instance.

30.3. RDS - Relational Database Service aws:rds

  • Instance type
  • Db storage options
    • aws:ebs
      • General SSD
      • Provisioned IOPS SSD
      • Magnetic

    • Aurora

      • MySQL and PostgreSQL- compatible db that is 5 times faster than standard MySQL
      • Distributed, fault-tolerant, self-healing that auto-scales up to 64TB per db instance
      • point-in-time recovery
      • continuous backup to S3
      • replication across 3 Availability Zones

30.4. EBS - Elastic Block Store aws:ebs

  • Volume types
    • General Purpose SSD
      • db
    • Provisioned IOPS SSD
      • low-latency for I/O intensive db workloads
    • Magnetic
      • low cost per gigabyte for backwards compatibility

30.5. ACM Certificate Manager aws:acm

Request a certificate
e.g. for media.a.com
  • Add CNAME _xxx.media.a.com _yyy.zzz.acm-validations.aws.

30.6. CloudFront aws:cloudfront

CloudFront URL without SSL
http://d111111abcdef8.cloudfront.net
(no term)
Default 24 hours
Distribution
Web or RTMP (Adobe)

  • General

    • Alternate Domain Names (CNAMES)
    • e.g. media.a.com then CNAME media d111111abcdef8.cloudfront.net
      SSL Certificate
      refer tod aws:acm
      Default Root Object
      e.g. index.html

  • Web

    Specify an origin
    e.g. AWS S3
    Origin Domain Name
    e.g. media.a.com.s3.amazonaws.com
    Origin Path
    empty
    Origin ID
    a distribution might have multiple origins e.g. s3-media.a.com
    Restrict Bucket Access
    d:No. Yes to force users to always use CloudFront URLs not the S3 URLs
    Origin Custom Headers (Header Name:Value)
    send these request headers to S3
    • Specifiy Cache Behavior
    (no term)
    Query String Forwarding and Caching
    None
    the same cache is served for media.a.com/b.jpg and media.a.com/b.jpg?v=1
    Forward all, cache based on whitelist
    for some query parameters
    • one parameter one line
    Forward all, cache based on all
    for all query parameters

30.7. S3 aws:s3

Alternative
Azure, Google Cloud, BackBlaze B2
Region code name
https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
(no term)
By default, the maximum number of buckets is 100
(no term)
Create bucket and enable logging in bucket doesn't cost anything
(no term)
Create a user in IAM with access key and assign the following policy and assign the policy
  • First create a group policy with ListAllMyBuckets permission then add a new user to that group

  • The user policy is Allow All for 2 resources arn:aws:s3:::<bucket_name> and arn:aws:s3:::<bucket_name>/* 

    {
"Effect": "Allow",
"Action": [
    "s3:ListAllMyBuckets"
],
"Resource": "arn:aws:s3:::*"
},
{
"Effect": "Allow",
"Action": [
    "s3:*"
],
"Resource": [
    "arn:aws:s3:::<bucket_name>",
    "arn:aws:s3:::<bucket_name>/*",
]
}
(no term)
Properties of the bucket
  • may set a tag with key application and value is the website name
(no term)
Domain

  • Manual

    • Add CNAME media s3.amazonaws.com as the same as the bucket name media.a.com. The following URLs will work
    • https://media.a.com/
    • https://media.a.com.s3.amazonaws.com/
    • https://s3.amazonaws.com/media.a.com/
      • Add robots.txt in bucket to prevent SE from indexing the files
    User-agent: *
Disallow: /
  • Refer to aws:cloudfront

30.7.1. Storage Classes

STANDARD: STANDARD_IA (infrequent class): Retrieval fee. Real-time retrieval. Good for larger objects (>128kb). GLACIER: Retrieval fee. Retrieval time is hours. REDUCED_REDUNDANCY: less durability and availability. Best for storing thumbnail images.

30.8. Trusted Advisor aws:trusted advisor

  • Real time guidance to help provision AWS resources
  • Checks
    • Cost Optimization
    • Performance
    • Security
    • Fault Tolerance
    • Service Limits

30.9. CodePipeline aws:codepipeline

31. Database

31.1. Basics

31.1.1. Parent vs child

Table A's PK is referenced as a FK field in another Table B
then Table A is parent to Table B or Table A has child relationship with Table B

31.1.2. CAP Theorem - Consistency, Availability & Partition Tolerance

Relational Database (RDBMS) is CA MongoDB and document-oriented db is CP

31.1.3. Vertical and Horizontal Scaling

Vertical scaling
increase capacity of a single server by adding RAM, CPU or storage space
Horizontal scaling
divide the dataset and load over multiple additional servers

31.1.4. Modeling Optimization

First normal form (1NF)
every field should have one value. (Not comma separated values)
Second normal form (2NF)
If you have multiple primary keys (composite keys) in a table, every field should be unique to all of those composite keys
  • e.g. CourseID and Date are 2 primary keys in table A. Field CourseTitle is unique to courseID. courseTitle should not be in table A. Create a new table B to include CourseID and CourseTitle
Third normal form (3NF)
Say you have CourseID, Date, Room, Seats in table A. Seats is unique to Room. You should create table B to hold Room, Seats and include just Room in table A

31.1.5. Integrity mode - ACID and BASE

ACID
usually what relational DB provides
Atomic
All parts involved should be either success or failure as a whole
Consistent
cannot result in a situation that violates any of the integrity rules
Isolated
all relevant data is locked until the transaction is finished. (Race condition)
Pessimistic Locking
Once locked, others can't even read (be refused or have to wait)
Optimistic Locking
Once one request lock data, other requests can read but roll back when try to write to db while lock is encountered
Durable
Once success, always success
BASE
usually NoSQL provides
  • Basic availability
  • Soft-state
  • Eventual consistency

31.1.6. Index

One table can have only one clustered index which is by default the primary key column. You may make another column as non-clustered index. Each non-clustered index creates a new table which includes the clustered index column and the new column (sorted). When a new row is added to the main table, a new row is also added to each non-cluster indexed table.

31.2. Access

31.2.1. Null

IsNull(col1) return true or false
Nz(col1,'') return '' if col1 is null

Sometimes IsNull() and Nz() don't work, e.g. UcanAccess, Use NVL() which is the same as Nz(). Or use col1 IS NULL

31.2.2. Coldfusion Datetime

<cfset dStart = DateFormat(form.startdate, "mm/dd/yyyy")>

SELECT *
FROM aTable
WHERE DateStart >= #CreateDate(Year(now()),Month(now()),Day(now()))#
AND DateEnd <= #CreateDate(Year(now()),Month(now()),DaysInMonth(now()))#
AND DateStart BETWEEN #dStart# AND #CreateDate(Year(dStart),Month(dStyart),DaysInMonth(dStart))#

31.2.3. De-dup

DELETE t1.*
FROM abc t1
WHERE t1.id NOT IN (
   SELECT TOP 1 id
   FROM   abc t2
   WHERE  t1.col1 = t2.col1
  AND t1.col2 = t2.col2
   ORDER BY t2.datefield DESC, t2.id )

31.2.4. Random Order

SELECT *
FROM aTable
ORDER BY rnd(INT(NOW*id)-NOW*id)

<!--- OR
rnd(aTable.ID)
--->

31.2.5. Troubleshooting

31.2.5.1. "Record is too large"

This is because the size of a single record is around 2K. You can change the data type for some of fields to Long Text (>Access 2013) or Memo (<= Access 2000)

31.2.5.2. "Application uses a value of the wrong type for the current operation"

It's possibly due to you are using Microsoft Access with Unicode driver in ColdFusion and you insert to a memo or long text field. cf_sql_longvarchar doesn't work. You should use <cfqueryparam cfsqltype="CF_SQL_CLOB" value="#description#"> instead.

31.3. MySQL

31.3.1. Version, Settings, Global variables

  • service mysql status
  • sudo systemctl restart mysql.service or service mysql restart or service mysqld restart
  • MySQL driver DSN
    • mysql:host=localhost;dbname=oophp
    • mysql:host=localhost;port=3307;dbname=oophp
31.3.1.1. Get settings, global variables, session variable
-- all settings / global variables
SHOW VARIABLES;

-- one setting
SHOW VARIABLES LIKE "%version%";

SHOW VARIALBES LIKE "%port%";

-- default character set for database 
-- MariaDB uses utf8mb4 as default charset
SHOW VARIABLES LIKE 'char%';

-- default collation `collation_database`
-- MriaDB default is utf8mb4_general_ci
SHOW VARIABLES LIKE 'collation%';

-- Or just use one line get 2 variables
SELECT @@character_set_database, @@collation_database;

-- @@var_name means to get the session value if exists and the global value otherwise
-- To get or set session value specifically
SELECT @@SESSION.character_set_database;
-- To get or set global value specifically
SELECT @@global.character_set_database;

-- SET default is to set session variable
SET my_session_system_var_name = 'a';
-- eq. to
SET @@my_session_system_var_name = 'a';
SET @@SESSION.my_session_system_var_name = 'a';
SET SESSION my_session_system_var_name = 'a';

-- SET global system variable
SET @@GLOBAL.my_session_system_var_name = 'a';
SET GLOBAL my_session_system_var_name = 'a';

-- @ is User-defined variable

-- default collation for a charset
-- utf8mb4_general_ci
-- refer to mysql:charset
SHOW CHARACTER SET WHERE CHARSET = 'utf8mb4';

-- check if it's MariaDB
SELECT VERSION();

-- Database table mysql
SELECT User, Host, Password FROM mysql.user;
31.3.1.2. Config: mysqld
  • 31.4.2
  • default which means it only allows localhost connection
  • allow all network access
  • MySQL 8.0 may not be present. Add it
    • Make sure to comment out skip-networking directives
  • default 1 since 5.6.6
31.3.1.3. MySQL Mode (sql_mode)
  • SELECT @@SESSION.sql_mode;
  • Values
    STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION
    default
    • STRICT_TRANS_TABLES
      • Enable strict mode for transactional storage engines
      • insert query should provide all values that are not null
      • It includes ERROR_FOR_DIVISION_BY_ZERO, NO_ZERO_DATE, NO_ZERO_IN_DATE
    • NO_ENGINE_SUBSTITUTION
      disabled
      CREATE TABLE uses default engine if not specified, and ALTER TABLE throws warning and the table is not altered
      enabled
      an error occurs and the table is not created or altered if the desired engine is unavailable
    (no term)
    ANSI_QUOTES
    enabled
    double quotes cannot be used to quote literal strings. Both " and ` are identifier quote character
31.3.1.4. Allow remote access to user root
  • mysql:plugin:auth_socket

  • Set priviledges 

    -- change 'password' to the password you want for the user root
-- '%' means all host, this will add a row in table `user` of database `mysql`
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'password';
FLUSH PRIVILEGES;
-- Then restart :: sudo systemctl restart mysql.service

    • May encounter `Your password does not satisfy the current policy requirements`

      • The best way is to uninstall the validate_password plugin as changing the config file to change the policy level to low is not successful
      • mysql -uroot -p and then
  • Set 2 to allow all network access
31.3.1.5. Minimum Config

http://www.tocker.ca/2014/03/10/configuring-mysql-to-use-minimal-memory.html https://github.com/levonlee/vagrant/tree/master/build/docker-compose/mysql 

[mysqld]
innodb_buffer_pool_size=5M
innodb_log_buffer_size=256K
query_cache_size=0
max_connections=10
key_buffer_size=8
thread_cache_size=0
host_cache_size=0
innodb_ft_cache_size=1600000
innodb_ft_total_cache_size=32000000

# per thread or per operation settings
thread_stack=131072
sort_buffer_size=32K
read_buffer_size=8200
read_rnd_buffer_size=8200
max_heap_table_size=16K
tmp_table_size=1K
bulk_insert_buffer_size=0
join_buffer_size=128
net_buffer_length=1K
innodb_sort_buffer_size=64K

#settings that relate to the binary log (if enabled)
binlog_cache_size=4K
binlog_stmt_cache_size=4K
31.3.1.6. charset, collation mysql:collation
  • Collation
    • A set of rules that defines how to compare and sort character strings
    • Every collation belongs to a single character set
    • A character set has at least one collation and most have 2 or more collations
    • case-insensitive at least in stopwords matching in full-text search
      *_cs
      case-sensitive
  • Character Set
    latin1
    default for MySQL, 1 byte
    utf8
    default collation utf8_general_ci, 3 bytes
    utf8mb4
    MariaDB default, default collation utf8mb4_general_ci, 4 bytes. PHP's UTF8 is 4 bytes
  • Get all available character sets and their default collation
    • SHOW CHARACTER SET;
    • Maxlen column shows each character byte

  • Get and set charset and collation 

    SELECT @@character_set_database, @@collation_database;
ALTER DATABASE db_name CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
ALTER TABLE {tbl_name} CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
ALTER TABLE {tbl_name} MODIFY "field_name" "field_type" CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
  • See MySQL Index
31.3.1.7. Memory, InnoDB Buffer, packet
name
default
innodb-buffer-pool-size
innodb-buffer-pool-chunk-size * innodb-buffer-pool-instaces
max-allowed-packet
16M, total BLOB data size for a single row
31.3.1.8. Last update time for a table in a database
SELECT UPDATE_TIME, TABLE_SCHEMA, TABLE_NAME
FROM   INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA NOT LIKE '%information_schema%'
AND TABLE_SCHEMA LIKE '%your_db_name%'
AND TABLE_NAME LIKE '%your_db_table_name%'

SELECT MAX(UPDATE_TIME), TABLE_SCHEMA
FROM   INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA NOT LIKE '%information_schema%'
GROUP BY TABLE_SCHEMA
ORDER BY MAX(UPDATE_TIME)
31.3.1.9. Query Log

MySQL >= 5.1.12 

show variables like 'general_log%';
// see if it's enabled
show variables like 'log_output%';
// TABLE :: log to a table, FILE or NONE

// Change this at runtime
SET GLOBAL log_output = 'TABLE';
SET GLOBAL general_log = 'ON';

SELECT * FROM general_log

// If you prefer to output to a file instead of a table:

SET GLOBAL log_output = "FILE"; the default.
SET GLOBAL general_log_file = "/path/to/your/logfile.log";
SET GLOBAL general_log = 'ON';
31.3.1.10. Threads
  • Before v8.0.14, 1 query uses only 1 CPU core (concurrency not parallism)
    • v8.0.14 uses multi CPU cores only in SELECT COUNT(*) FROM t1 without any WHERE nor GROUP BY
  • Concurrency threads can be configured

    • See settings 

      SELECT @@innodb_thread_concurrency, @@innodb_read_io_threads, @@innodb_write_io_threads;
    • default 0 means no limit. Any non-zero throttles
    • default 4
    • default 4
31.3.1.11. sql_mode
-- Remove a mode from sql_mode which is comma delimited
SET SESSION sql_mode=(SELECT REPLACE(@@sql_mode,'ONLY_FULL_GROUP_BY',''));

31.3.2. mysql Client

31.3.2.1. Commands
  • https://dev.mysql.com/doc/refman/8.0/en/mysql-commands.html
  • status
    • Current user, database, connection port, uptime, character set, MySQL Server version, command line delimiter
  • display the result using vertical format
  • open and edit query/command buffer
    • After exit the buffer, type the delimiter ; and enter to run the command in the buffer
  • print the current query/command buffer
31.3.2.2. mysql Client Options
  • https://dev.mysql.com/doc/refman/8.0/en/mysql-command-options.html

  • Execute 

    mysql -e "SELECT * FROM a WHERE col1 LIKE '%some%'" -uroot -p mydbname > /path/to/bk.tsv
    Escape " inside the query
    \"
    Escape \
    \\
    (no term)

    Use Heredoc 

    mysql -uroot -pABC mydb <<EOFSQL
-- Avoid using backticks, escape with \`

EOFSQL
  • don't print the row of columns
  • batch mode. Print results using tab as column separator and new line as row separator (nontabular output). Default is column separator is | with spaces padded and new line as row separator (tabular output)
  • For nontabular output, special characters are escaped. Disable this character escaping
    Newline
    \n
    Tab
    \t
    NUL
    \0
    backslash
    \\

31.3.3. SSH Tunneling

# forward local port 9001 to mysshhost port 3306, login to the mysshhost shell
ssh -L 9001:localhost:3306 mysshhost

# In anothe shell
mysql -uroot -p -P 9001 --protocol=tcp mydb

# or use -h, but it cannot be localhost!
mysql -uroot -p -P 9001 -h 127.0.0.1 mydb

# Don't need SSH Tunneling for dump, the output is redirected to local backup.sql
ssh mysshhost mysqldump -uroot -p [..options] mydb > backup.sql

31.3.4. Install

# Ubuntu
apt-get update; apt-get install mysql-server
service mysql start
# or
sudo systemctl restart mysql.service

# Debian
yum install php-mysql
service httpd restart
service mysqld start

# both
mysql_secure_installation

# status
systemctl status mysql.service
31.3.4.1. Manage users
  • Refer to mysql:user:remote access
  • auth_socket usually works in Linux where auth is done through Linux localhost users auth
    • In Ubuntu, user root by default uses auth_socket instead of mysql_native_password. Later is required for external db connection using user root
      • ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'my-strong-password';
      • FLUSH PRIVILEGES;
-- see a list of all users
SELECT user, host, password, authentication_string, plugin FROM mysql.user;

CREATE USER 'sammy'@'localhost' IDENTIFIED BY 'password';

-- create a user that can connect to this server from any external IP address
CREATE USER 'sammy'@'%' IDENTIFIED BY 'password';

-- CREATE USER 'sammy'@'192.168.100.%%' IDENTIFIED BY 'password';

-- grant all privileges to one database
GRANT ALL PRIVILEGES ON mydatabase.* TO 'sammy'@'localhost';

-- same as above but the new user can also grant privileges to others
GRANT ALL PRIVILEGES ON mydatabase.* TO 'sammy'@'localhost' WITH GRANT OPTION;

-- grant all privileges to all tables/databases
GRANT ALL PRIVILEGES ON *.* TO 'sammy'@'localhost' WITH GRANT OPTION;

-- except for granting privileges for a brand new user, after INSERT, UPDATE, DELETE, flush is required
FLUSH PRIVILEGES;

-- show granted privileges
SHOW GRANTS FOR 'sammy'@'localhost';
SHOW GRANTS;
-- eq. to 
-- SHOW GRANTS FOR CURRENT_USER();
-- SHOW GRANTS FOR CURRENT_USER;
-- Show grants for all users
select * from information_schema.user_privileges;

-- remove a user
DELETE FROM mysql.user WHERE user = 'sammy';
31.3.4.2. revoke 
REVOKE INSERT ON *.* FROM 'li'@'localhost';

REVOKE ALL, GRANT OPTION FROM 'li'@'localhost';

-- it does not remove the user from mysql.user table
-- DROP USER :: remove one or more user accounts and all grants of them

SHOW GRANTS;
31.3.4.3. Create database
CREATE DATABASE `dbname_in_lowercase`;
USE `dbname_in_lowercase`;
SHOW TABLES;
31.3.4.4. MySQL does not start after installation, Ubuntu

No directory, logging in with HOME=/ and service mysql status shows it is inactive. This is new.. Change the owner:group for relevant folders 

chown -R mysql:mysql /var/lib/mysql /var/run/mysqld
31.3.4.5. Reset root password, Ubuntu

ERROR 1045: Access denied for user: 'root@localhost' (Using password: NO) or yes 

# stop
sudo /etc/init.d/mysql stop

# skip permission check
sudo /usr/sbin/mysqld --skip-grant-tables --skip-networking &

# start mysql client without a password
mysql -u root
FLUSH PRIVILEGES;

# Update root password
SET PASSWORD FOR root@'localhost' = PASSWORD('password');

# If you have a mysql root account that can connect from everywhere, you should also do:
UPDATE mysql.user SET Password=PASSWORD('password') WHERE User='root';

# Alternate
USE mysql
UPDATE user SET Password = PASSWORD()
WHERE Host = 'localhost' And User='root';

# Alternate for root can access from everywhere
USE mysql
UPDATE user SET Password = PASSWORD('password');
WHERE Host = '%' AND User = 'root';

FLUSH PRIVILEGES;
exit
service mysql restart

# or

sudo /etc/init.d/mysql stop
sudo /etc/init.d/mysql start

31.3.6. Data Types

31.3.6.1. Integer
NOT NULL default
0
SIGNED
contains negative, 0 and positive
UNSIGNED
contains 0 and positive
TINYINT
  • 1 byte
  • -128 to 127
  • 0 to 255
  • m is just display-column width. Storage size is the same
  • PHP always returns the same range as TINYINT
(no term)
SMALLINT
  • 2 bytes
  • -32,768 to 32,767
  • 0 to 65,535
(no term)
MEDIUMINT
  • 3 bytes
  • -8,388,608 to 8,388,607
  • 0 to 16,777,215
(no term)
INT
  • 4 bytes
  • -2,147,483,648 to 2,147,483,647
  • 0 to 4,294,967,295
    • The max positive number to Unix Timestamp is Sun Feb 07 2106 06:28:15 GMT+0000
  • m is just display-column width. Storage size is the same
(no term)
BIGINT
  • 8 bytes
  • -2^63 to 2^63-1
    9.223x10^19
    9.223MMM
  • 0 to 2^64-1
    18.447x10^19
    18.447MMM
31.3.6.2. Decimal
  • DECIMAL(M,D)
    • In MySQL, NUMERIC(M,D) is exactly the same as DECIMAL(M,D)
    • default 10. Max 65. total number of digits
    • default 0. Max 30. number of digits to the right of the decimal point. It should not be larger than M
  • Range of -999 to 999, 3 digits, total 2 bytes
  • 9 digits on either side of the decimal point (the scale). Total (2*9)/9*4 = 8 bytes
  • 14 digits (9+5) on the left, 6 digits on the right. Total (4+3) + (3) = 10 bytes
  • 18 digits (9+9) on the left, 2 digits on the right. Total (4+4) + 1 = 9 bytes
  • Up to trillion dollars. 9+4 on left and 2 digits on right. 4+2 + 1 = 7 bytes
  • Every 9 digits is 4 bytes, the remaining digits left over require:
    1 to 2 digits left over
    1 byte
    3 to 4
    2 bytes
    5 to 6
    3 bytes
    7-9
    4 bytes

  • Insert or Update a decimal field with empty string 

    -- Throws a warning but the row is inserted with value 0.00
INSERT INTO myTable (decimal_field) VALUE ('');
  • UNSIGNED for float, double and decimal are deprecated MySQL 8+.

    • After upgraded to MySQL 8+ 

      -- remove unsigned
alter table my_table
    modify my_col decimal(20, 2) default null;

alter table my_table
  add constraint my_col_unsigned
  check (my_col > 0);
31.3.6.3. BOOL and BOOLEAN
31.3.6.4. Datetime mysql:datatype:datetime

  • Get global and sessiond timezones 

    SELECT @@GLOBAL.time_zone, @@SESSION.time_zone;

-- List of all timezones
SELECT * FROM mysql.time_zone_name;

    If it's SYSTEM, refer to linux:timezone

  • format 'HH:MM:SS' from '-838:59:59.000' to '838:59:59.000'
  • return string e.g. 2019-07-12 01:02:03 in session timezone
  • returns a string YYYY-MM-DD hh:mm:ss or numeric YYYYMMDDhhmmss depending on the context
    fsp argument
    a fractional seconds precision from 0 to 6
    (no term)
    In session timezone
  • Date comparison
    1 Year ago
    post_date > DATE_SUB(NOW(), INTERVAL 1 YEAR)
  • return integer or float or NULL. Convert the string of time in session timezone to UTC integer
    date
    maybe a
    • DATE
    • DATETIME
    • TIMESTAMP string, see NOW()
      UNIX_TIMESTAMP('2024-01-19T23:03:48.676Z')
      1705705428.676
    • a number in YYMMDD, YYMMDDhhmmss, YYYYMMDD, or YYYYMMDDhhmmss format
-- Custom datetime
TIMESTAMP('2003-12-31')

CAST(post_date AS DATE) > '2014-01-01'
FROM_UNIXTIME(intTime) > '2014-01-01'
FROM_UNIXTIME(intTime) BETWEEN '2014-01-01 00:00:00' AND '2014-12-31 23:59:00'

SELECT NOW(); -- datetime type

-- UNIX Time
-- default date format
DATE_FORMAT(FROM_UNIXTIME(intTime),'%Y-%m-%d %H:%i:%s')
UNIX_TIMESTAMP(NOW()) -- returns integer, accepts date type

-- both using timezone name, requires MySQL to support the timezone name
SELECT UNIX_TIMESTAMP(CONVERT_TZ('2022-04-01 11:50:00', 'America/Toronto', 'UTC'));

-- one using timezone name, requires MySQL to support the timezone name
SELECT UNIX_TIMESTAMP(CONVERT_TZ('2022-04-01 11:50:00', '-04:00', 'UTC'));

-- Guaranteed to work
-- The input date is of time zone '-04:00'. Convert timezone from '-04:00' to session timezone
SELECT CONVERT_TZ('2022-04-01 11:50:00', '-04:00', @@SESSION.time_zone);
-- eq.
SELECT CONVERT_TZ('2022-04-01 11:50:00', '-04:00', '+00:00');
-- eq. if session timezone is equal to system (e.g. Linux that MySQL is installed on) timezone
SELECT CONVERT_TZ('2022-04-01 11:50:00', '-04:00', 'SYSTEM');

SELECT * FROM a WHERE aUnixTimeColumn > UNIX_TIMESTAMP(NOW() - INTERVAL 24 HOUR) -- last 24 hours

-- From UNIX timestamp (int) to a specific timezone, return date type
SELECT CONVERT_TZ(FROM_UNIXTIME(aColumnOfUnixTimeStamp), 'UTC', '-04:00');
31.3.6.5. String Functions
  1. Regex search 
    -- column is integer
SELECT field
FROM myTable
WHERE field REGEXP '^-?[0-9]+$';
  2. REGEXP_REPLACE

    • Do more complicated find and replace. MariaDB driver on host is required 

      SELECT REGEXP_REPLACE("stackoverflow", "(stack)(over)(flow)", '\\2 - \\1 - \\3');
  3. String delimited list 
    substring_index ( substring_index ( context,',',1 ), ',', -1) ) 
substring_index ( substring_index ( context,',',2 ), ',', -1) ) 
substring_index ( substring_index ( context,',',3 ), ',', -1) ) 
substring_index ( substring_index ( context,',',4 ), ',', -1) )
-- it means 1st value, 2nd, 3rd, etc.
    Explanation
    original string is "34,7,23,89", substring_index( context,',', 3) returns "34,7,23". The outer substring_index takes the value returned by the inner substring_index and the -1 allows you to take the last value. So you get "23" from the "34,7,23"
    (no term)
    See mysql:string:list:temp table for more
  4. FIELD(str, str1, str2, ...)
    • return index position (starting from 1) of str in the str1, str2, ... list
      • 0 if not found or str is NULL
    • If all arguments including str are strings, then compare as strings. If all arguments are numbers, compare as numbers
      • Otherwise, compare as double
    • A good place to use it is in ORDER BY
    • The complement is ELT()
      • ELT(N, str1, str2, ... )
        SELECT ELT(1, 'Aa', 'Bb', 'Cc', 'Dd')
        'Aa'
  5. CONCAT(str1, str2, ...)

    • If there is any value that is NULL, CONCAT returns NULL 

      SET @s3 = 'http://s3.amazonaws.com/abc/';
SELECT CONCAT('t1', null); -- null
SELECT REGEXP_REPLACE(thumbnail.uri, "^(s3://)(.*)", CONCAT(@s3,'\\2')) AS thumbnail';

    • Multiple columns all have to have values that are not empty string nor null 

      SELECT * FROM aTable WHERE 1 = 1 AND
CONCAT(IF(TRIM(colA), 1, NULL),
       IF(TRIM(colB), 1, NULL),
       IF(TRIM(colC), 1, NULL));
    • See 31.3.18.1
    • See FIND_IN_SET()
  6. CONCAT_WS(separator, str1, str2, ...)

    • Concatenate columns or strings of one row. It doesn't skip empty string but it does skip NULL values 

      CONCAT_WS(',', colA, colB, 'Column C Value')
WHERE url_alias.source = CONCAT('node/', n.nid)

CONCAT_WS(',', NULL, '123', NULL) -- 123
CONCAT_WS(',', NULL, NULL) -- empty string
CONCAT_WS(',', NULL, '123', '', NULL) -- 123,,
CONCAT_WS(',', NULL, '123', TRIM(NULLIF(' ', '')), NULL) -- 123
  7. Replace all occurances, case-sensitve. multibyte safe 
    SELECT REPLACE('www.mysql.com', 'w', 'Ww');
  8. LIKE
    • expr LIKE pattern [ESCAPE 'escape_char']

      • Case insensitive. String comparisons are not case-sensitive unless one of the operands is case-sensitive (uses a case-sensitive collation or is a binary string) 

        SELECT 'abc' LIKE 'ABC';
--        -> 1
SELECT 'abc' LIKE _utf8mb4 'ABC' COLLATE utf8mb4_0900_as_cs;
--        -> 0
SELECT 'abc' LIKE _utf8mb4 'ABC' COLLATE utf8mb4_bin;
--        -> 0
SELECT 'abc' LIKE BINARY 'ABC';
--        -> 0
      • default \
      • expr NOT LIKE pat [ESCAPE 'escape_char'] eq. NOT (expr LIKE pat [ESCAPE 'escape_char'])
    • eq. *
      Ends with
      SELECT * FROM aTable WHERE colA LIKE '*.pdf'
    • like ?
    • use \
      \n
      newline character. \\n to escape \n
      \
      Use \\\\ to escape, if the target \ is at the end of the pattern string, use \\
  9. SOUNDDEX(), SOUNDS LIKE
  10. SUBSTRING()
    • alias SUBSTR
    SUBSTR(str,pos)
SUBSTR(str FROM pos)
SUBSTR(str,pos,len)
SUBSTR(str FROM pos FOR len)

-- The first 10 characters of a column
LEFT(aColumn, 10) =  SUBSTRING(aColumn, 1, 10)

-- The last 10 characters of a column
RIGHT(aColumn, 10) = SUBSTRING(aColumn, -10)

SUBSTRING('foobarbar' FROM 4) = SUBSTRING('foobarbar', 4) = 'barbar'

-- Get 6 characters from the 5 + 1 position
SUBSTRING('Quadratically',5,6) = SUBSTRING('Quadratically' FROM 5 FOR 6) = 'ratica'
  11. LOCATE()
    • multibyte safe and is case-sensitive only if at least one argument is a binary string
    SELECT LOCATE('bar', 'foobarbar') = 4;
SELECT LOCATE('xbar', 'foobar') = 0;

-- search 'bar' from 5 + 1 position
SELECT LOCATE('bar', 'foobarbar', 5) = 7;
  12. MATCH … AGAINST
    • https://dev.mysql.com/doc/refman/8.0/en/fulltext-search.html
    • Can be used in SELECT, GROUP BY, HAVING or ORDER BY
    • expr in AGAINST is a search string which must be a string value that is constant during query evaluation. This rules out, for example, a table column because that can differ for each row
    • On top of stopwords, words that have length less than SELECT @@innodb_ft_min_token_size = 3; and larger than SELECT @@innodb_ft_max_token_size = 84; are not in the fulltext index
      • Do not apply to FULLTEXT indexes created using the ngram parser. ngram token size is defined by the ngram_token_size option
      • Do not apply to the search string with leading plus and minus sign '+word +the'
    • Returns the matching score, the higher the more relevant the matching is
    MATCH (col1,col2,...) AGAINST (expr [search_modifier])

search_modifier:
  {
   IN NATURAL LANGUAGE MODE
     | IN NATURAL LANGUAGE MODE WITH QUERY EXPANSION
     | IN BOOLEAN MODE
     | WITH QUERY EXPANSION
  }
    1. FULLTEXT index type
      • Only for CHAR, VARCHAR or TEXT columns
      CREATE FULLTEXT INDEX myfulltextindex ON mytable (field1, field2, field3);
    2. Stopwords
      • Stopwords are not part of full text index and are ignored in case of a linguistic search
      • Default, it means to use stopwords from this order
        • SELECT @@innodb_ft_user_stopword_table
        • SELECT @@innodb_ft_server_stopword_table
        • Default set of stopwords
    3. Boolean Mode
      AGAINST ('+WordMustHave -WordMustNotHave WordNiceToHave' IN BOOLEAN MODE);
  13. FIND_IN_SET
    • strlist is a comma separated string
    • Return
      0
      if str not in strlist
      null
      if either str or strlist is NULL
    FIND_IN_SET(str, strlist)

-- 2
SELECT FIND_IN_SET('b', 'a,b,c,d');

-- 0
SELECT FIND_IN_SET('b', 'a,b ,c,d');
SELECT FIND_IN_SET('b', 'a, b,c,d');

-- Change delimiter
SELECT FIND_IN_SET('b', REPLACE('a|b|c|d', '|', ','));

-- Add to list (requires column users.groups is non-null)
UPDATE users
SET groups = CONCAT(groups, '|account')
WHERE userid = 1234
      AND FIND_IN_SET('account', REPLACE(groups, '|', ',')) = 0;

-- Remove from list (remove 2 from comma-delimited list, and list can contain multiple 2's)
-- - Requires columnn mytable.categories is non-null
UPDATE mytable
SET categories =
        TRIM(BOTH ',' FROM
             REPLACE(
                     REPLACE(
                             CONCAT(
                                     ',',
                                     REPLACE(categories, ',', ',,'),
                                     ','
                             ),
                             ',2,',
                             ''
                     ),
                     ',,',
                     ','
             )
        )
WHERE FIND_IN_SET('2', categories)
31.3.6.6. CHAR and VARCHAR
  • 1 character 1 byte
  • CHAR(n)
    • Fixed byte
    • Required no storage space. So CHAR(n) is n bytes
    • When retrieved, trailing spaces are removed unless PAD_CHAR_TO_FULL_LENGTH mode is enabled
      • Trailing spaces are not retained when fetched!
    • right padding with space
  • VARCHAR(n)
    • Variable length
    • n is called column length and can be from 0 to 65,535
    • Not padded thus trailing spaces are retained
      • VARCHAR(3) and insert a column length 4 value and with trailing spaces, the value (the trailing spaces) will be truncated and saved. If the value has no trailing spaces, it will throw an error
    • 1-byte or 2-byte length prefix plus actual data
      • Length prefix
        • If actual data =< 255 bytes, 1 storage byte
        • if actual data > 255, 2 storage byte
      • see 31.3.1.6
31.3.6.7. TEXT Type
  • TEXT data is not stored in the database server’s memory. Quering TEXT data requires reading from disk, much slower than CHAR and VARCHAR
  • It holds data that is nonbinary strings
  • It doesn't count towards the maximum row size for each record
  • No padding on insert and no bytes are stripped on select
    • If it is indexed, index entry comparisons are space-padded at the end
  • Insert with exceeding max length will be truncated
  • Cannot have DEFAULT value
  • 255 characters, 255 Byte
  • 65,535 characters, 64kb
  • 16,777,215, 16mb
  • 4,294,967,295 characters, 4GB
31.3.6.8. Blob
  • Binary large object for variable amount of data
  • Holds binary strings or so called byte strings
  • Data is stored on hard disk rather than in memory
  • TINYBLOB
  • BLOB
  • MEDIUMBLOB
  • LONGBLOB
31.3.6.9. Null mysql:datatype:null
IFNULL(column_name, 0)
WHERE column_name IS NOT NULL
31.3.6.10. UUID and GUID mysql:uuid

MySQL 8.0 adds a few functions 

CREATE TABLE t (id binary(16) PRIMARY KEY);

INSERT INTO t VALUES(UUID_TO_BIN(UUID()));

SELECT BIN_TO_UUID(id) FROM T;

-- the records are inserted in random locations in the index tree which impacts performance
-- To insert records in order, better to do it like this

INSERT INTO t VALUES(UUID_TO_BIN(UUID(), true));

MySQL 5.6

  • Use mariadb:uuid instead of the following
  • Treat the following as the last resort
  • The following is not performance optimized
CREATE TABLE users(id_bin binary(16), name varchar(200));
INSERT INTO users VALUES (UNHEX(REPLACE(UUID(),'-','')), 'Andromeda');

-- id_text is a virtual column and doesn't occupy disk

CREATE TABLE users(
id_bin binary(16),
id_text varchar(36) GENERATED always as
 (insert(
    insert(
  insert(
    insert(hex(id_bin),9,0,'-'),
    14,0,'-'),
  19,0,'-'),
    24,0,'-')
 ) virtual,
name varchar(200));

INSERT INTO users (id_bin,name)
  VALUES(UNHEX(REPLACE(UUID(),'-','')), 'Andromeda');

-- If id_text is not an index, don't use WHERE ~WHERE id_text=text_form_of_XYZ~
-- If id_text is made to an index column, then it occupies space
ALTER TABLE users ADD UNIQUE(id_text);

31.3.7. Index, Composite Index

  • Index columns used in WHERE, ORDER BY and GROUP BY clauses
  • A column with <= 767 bytes can be indexed in Antelope file format which is default for MySQL 5.6 or lower
    • varchar(255) in utf8 charset max byte size 255*3=765
    • varchar(191) in utf8mb4 charset max byte size 191*4=764

    • Barracuda file format and DYNAMIC row format become the default since MySQL 5.7.7, which allows index key prefixes up to 3072 bytes (Barracuda + COMPRESSED row format can also reach)

      • becomes available on mysqld (MySQL) server version > 5.5.14 and default since 5.7.7

      • Use these settings in my.cnf and restart MySQL 

        [mysqld]
# innodb_large_prefix is to be deprecated. True to use larger index prefix length than 767 bytes
innodb_large_prefix=true
innodb_file_format=barracuda
innodb_file_per_table=true
      • As of Nov 25, 2016, Pantheon MySQL (innodb) version is 5.6.26 and does not have the above setting
      • If changing MySQL server setting is not possible and the index column has to be utf8mb4, then need to convert those columns in every table to have varchar(191). Which is what WordPress does
      ALTER TABLE tbl_name CHANGE index_col index_col VARCHAR(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
  • See Index Hint
  • Some MySQL 8 tricks to troubleshoot
31.3.7.1. Composite index
  • An index on multiple columns. MySQL supports up to 16 columns
    • also called multiple column index

    • Table definition 

      CREATE TABLE table_name (
       c1 data_type PRIMARY KEY,
       c2 data_type,
       c3 data_type,
       c4 data_type,
       INDEX index_name (c2,c3,c4)
);

    • Or create index directly 

      CREATE INDEX index_name 
ON table_name(c2,c3,c4);
  1. Use leftmost prefix to identify whether a combo is optimized
    • Applies also for GROUP BY AND DISTINCT
      • Only these aggregated functions are allowed
        • MIN() and MAX() with or without GROUP BY
        • With no GROUP BY
          AVG(DISTINCT)
          one single argument
          SUM(DISTINCT)
          one single argument
          COUNT(DISTINCT)
          multiple column arguments
      • Constants must be used and operator is =
        • Range like > causes the index to be partially used which is called Tight Index Scan.
          • Both Loose Index Scan (better) and Tight Index Scan (good) prevent creating temporary tables

          • Filling gaps in the leftmost prefix is another way to achieve Tight Index Scan 

            -- There is a gap in the GROUP BY, but it is covered by the condition c2 = 'a':
SELECT c1, c2, c3 FROM t1 WHERE c2 = 'a' GROUP BY c1, c3;

-- The GROUP BY does not begin with the first part of the key, but there is a condition that provides a constant for that part:
SELECT c1, c2, c3 FROM t1 WHERE c1 = 'a' GROUP BY c2, c3;
      • If Loose Index Scan is applicable, EXPLAIN will show
        • Using index for group-by in the Extra column
        • Or in JSON using_index_for_group_by
    • Index on 3 columns c2, c3, c4
      • optimize queries using these combinations given that operator = is only used
        • c2
        • c2, c3
          • WHERE c2 = 1 AND c3 = 3
          • WHERE c2 = 1 AND (c3 = 3 OR c3 = 4)
        • e.g.
          • WHERE c2 = 1 AND c3 = 3 AND c4 = 4
      • Notice, not optimized for the following
        • WHERE c3 = 3
      • Index is not fully used
        • WHERE c2 = 2, c4 = 4
          • The index is used up to c2
        • WHERE c2 = 2 AND c3 > 3 AND c4 = 4
          • operator is > (range), so the only c2 is used for the index
      • c2 is the most selective column meaning given c2, we can get more unique records
      • Redundant indexes. We have an existing composite index c2, c3, c4. The following are redundant indexes
        • c2
        • c2, c3
31.3.7.2. Index Hint
  • Index hints can be used in SELECT queries to specify whether or not to use the index column
    • https://dev.mysql.com/doc/refman/8.0/en/index-hints.html
    • With no index hint, MySQL will smartly decided whether to force using index
    • In fact, the less the number of query matched records are, the faster the speed is. Because for each selected record, MySQL has to bind the indexed column to the result
  • Each hint requires index names, not column names
    • SHOW INDEX FROM myTable
      • This also shows the cardinality of each index
        • Estimated number of uniquevalues in the index.
        • The higher, the greater the chance that the query optimizer uses the index for lookups
  • Index hints apply only to SELECT and UPDATE statements
  • The FORCE INDEX hint acts like USE INDEX (index_list), with the addition that a table scan is assumed to be very expensive. In other words, a table scan is used only if there is no way to use one of the named indexes to find rows in the table.

  • Syntax 

    tbl_name [[AS] alias] [index_hint_list]

index_hint_list:
index_hint [index_hint] ...

index_hint:
USE {INDEX|KEY}
  [FOR {JOIN|ORDER BY|GROUP BY}] ([index_list])
  | {IGNORE|FORCE} {INDEX|KEY}
  [FOR {JOIN|ORDER BY|GROUP BY}] (index_list)

index_list:
index_name [, index_name] ...

  • Samples 

    SELECT * FROM table1 USE INDEX (col1_index,col2_index)
  WHERE col1=1 AND col2=2 AND col3=3;

SELECT * FROM table1 IGNORE INDEX (col3_index)
  WHERE col1=1 AND col2=2 AND col3=3;

SELECT * FROM t1 USE INDEX (i1) IGNORE INDEX FOR ORDER BY (i2) ORDER BY a;

31.3.8. Common queries, functions, operators

31.3.8.1. Reuse SELECT alias
  • https://dev.mysql.com/doc/refman/8.0/en/select.html
  • SELECT aliases (or column aliases) can only be used in later columns in the same SELECT, ORDER BY, GROUP BY and HAVING, not WHERE!
    • Order
      • FROM
        • SELECT aliases cannot be used
      • WHERE
        • SELECT aliases cannot be used
      • SELECT select_expr1 AS myAlias1, select_expr2 AS myAlias2
        • Aliases can be used in later GROUP BY, HAVING and ORDER BY with some exceptions
          • User-defined variables
            • The order of evaluation for expressions involving user variables is undefined. For example,
              • there is no guarantee that SELECT @a, @a:=@a+1 evaluates @a first and then performs the assignment.

        • An earlier alias can be used in later select_expr 

          SELECT
s.f1 + s.f2 AS total_sale, 
s.f1 / (SELECT total_sale) AS f1_percent
IF((SELECT total_score) > 100, 1, 0) AS good_or_bad,
IF((SELECT good_or_bad), 'good score', 'bad score') AS good_or_bad_text,

-- Limitation of this method ~(SELECT my_alias)~ : my alias should not be a group/aggregation function result. e.g. SELECT COUNT(*) AS localCount, (SELECT localCount) + 1
        • Aliases can be passed onto only the first subquery not to the next deeper level subqueries in later select_expr
          • Use JOIN
      • GROUP BY
      • HAVING
        • Aggregated values from the SELECT
        • Keys from the GROUP BY
        • Functions that take no argument e.g. current date
        • Expressions composed of the above
        • Besides only the above, MySQL has an extension to standard SQL:
          • permits HAVING to refer to columns in the SELECT list and columns in outer subqueries as well.
      • SELECT
      • ORDER BY
        • select_expr alias can be used, it's picked first before FROM table.aCol
      • LIMIT
        • Constants
        • Local variables
  • In HAVING
    • PhpStorm might report the following if an alias of aggregate function result is used
      • Using aggregate-free condition(s) in HAVING clause might be inefficient. Consider moving them to WHERE
  1. SELECT alias CANNOT be used in WHERE and FROM, need subquery
    Use sub query
    create a sub table with additional computed fields added in original table, and select from it later
    SELECT tbl1.total_sale,
   tbl1.f1 / tbl1.total_sale AS f1_percent
FROM (SELECT s.f1 + s.f2 AS total_sale,
         s.f1
  FROM sales s) AS tbl1;
31.3.8.2. Select all columns except some columns
SET @sql = CONCAT(
    'SELECT ',
    (SELECT REPLACE(GROUP_CONCAT(COLUMN_NAME), '<columns_to_omit>,', '')
     FROM INFORMATION_SCHEMA.COLUMNS
     WHERE TABLE_NAME = '<table>'
       AND TABLE_SCHEMA = '<database>'),
    ' FROM <table>');
PREPARE stmt1 FROM @sql;
EXECUTE stmt1;
31.3.8.3. IN 
SELECT userid, created_by FROM users WHERE userid in ($x);
-- eq.
SELECT userid, created_by from users WHERE $x in (userid);

SELECT userid FROM users WHERE login IN ($x) OR dispname OR ($x)
-- eq.
SELECT userid FROM users WHERE $x IN (login,dispname)
31.3.8.4. JOIN
  • In MySQL, JOIN, CROSS JOIN, and INNER JOIN are the same.
    • In standard SQL, they are not equivalent. INNER JOIN is used with an ON clause, CROSS JOIN is used otherwise.
      • CROSS JOIN does not have ON
      • Table A of n rows CROSS JOIN Table B of m rows will result n*m rows
  • Comma separated tables is eq. to JOIN but the precedence is less than JOIN, INNER JOIN, LEFT JOIN, CROSS JOIN
    • Generally, avoid mixing comma separated tables with * JOIN *

  • USING vs ON 

    a LEFT JOIN b USING (c1, c2, c3)
a LEFT JOIN b ON a.c1 = b.c1 AND a.c2 = b.c2 AND a.c3 = b.c3

-- SELECT *
-- USING is
COALESCE(a.c1, b.c1), COALESCE(a.c2, b.c2), COALESCE(a.c3, b.c3)
-- ON is
a.c1, a.c2, a.c3, b.c1, b.c2, b.c3
  • JOIN subquery
    • The only place to pass any parent columns is in ON. In other words, the subquery result has to be determined and then we can join this subquery
  1. Join on one row only
    • Tables
      • users
        • id
      • widgets
        • id
        • one to many
    -- In general, using ORDER BY ... LIMIT 1 is slower compared to MAX() or MIN()
SELECT *
FROM users JOIN (

    SELECT * FROM widgets
    WHERE id IN (
    SELECT MAX(id) FROM widgets
    GROUP BY user_id
    )

) AS most_recent_user_widget 
    ON users.id = most_recent_user_widget.user_id

-- Multiple columns sort
SELECT *
FROM users JOIN (

    SELECT * FROM widgets
    WHERE CONCAT_WS(0x1F, col2, id) IN (
    SELECT MAX(CONCAT_WS(0x1F, col2, id)) FROM widgets
    GROUP BY user_id
    )

) AS most_recent_user_widget 
    ON users.id = most_recent_user_widget.user_id

-- Group by but order by non-grouped columns

SELECT t1.*,  t2.final_exam_score
FROM (
  -- Original query
  SELECT scid, SUM(score) FROM s_score GROUP BY scid
) t1 JOIN (
-- Join on 1 row
SELECT *
FROM s_score
WHERE CONCAT_WS(0x1F, final_exam_score, ssid) IN (
     SELECT MAX(CONCAT_WS(0x1F, final_exam_score, ssid)) FROM s_score
     GROUP BY scid
  )
) t2
ON t1.scid = t2.scid;

-- Faster
SELECT t1.*,  t2.final_exam_score
FROM (
  -- Original query
  SELECT scid, SUM(score), MAX(CONCAT_WS(0x1F, final_exam_score, ssid)) AS _sortColumns FROM s_score GROUP BY scid
) t1 JOIN (
-- Join on 1 row
SELECT *
FROM s_score
) t2
ON t1._sortColumns = CONCAT_WS(0x1F, t2.final_exam_score, t2.ssid);
  2. Join on the 2nd, 3rd, 4th row

    • MySQL 8.0 can use ROW_NUMBER https://www.mysqltutorial.org/mysql-window-functions/mysql-row_number-function/ 

      ROW_NUMBER() OVER (<partition_definition> <order_definition>)

partition_definition:
PARTITION BY <expression>,[{,<expression>}...]

order_definition:
ORDER BY <expression> [ASC|DESC],[{,<expression>}...]

SELECT 
       productLine,
       productName,
       quantityInStock,
       ROW_NUMBER() OVER (
          PARTITION BY productLine 
          ORDER BY quantityInStock DESC) row_num
    FROM 
       products

    • In MySQL 5, to emulate the ROW_NUMBER 

      -- For each filtered log, find the 3rd, 2nd or 1st action date in the raw log conditionally depening on filtered log totalComms as the charge date
SELECT T1.*, _FindChargeDate._chargeDate
FROM filteredlog T1
LEFT JOIN (
        SELECT _RawLog.actiondate AS _chargeDate, CASE
                WHEN _RawLog.action IN (1, 2, 5) THEN 1 -- Action Type 1
                WHEN _RawLog.action IN (3, 4, 6) THEN 2 -- Action Type 2
                ELSE 0
            END AS _actionType,
               _RawLog.col1, _RawLog.col2, _RawLog.col3, _RawLog.col4,
            @rowNumberCalculated := CASE
                WHEN @rowNumberHashCalculated = CONCAT_WS('|', _RawLog.col1, _RawLog.col2, _RawLog.col3, _RawLog.col4, (SELECT _actionType))
                THEN @rowNumberCalculated + 1
                ELSE 1
            END AS _rowNumber,
               @rowNumberHashCalculated := CONCAT_WS('|', _RawLog.col1, _RawLog.col2, _RawLog.col3, _RawLog.col4, (SELECT _actionType))
        FROM rawLog AS _RawLog, 
             (SELECT @rowNumberHashCalculated := '', @rowNumberCalculated := 0) AS _rowRank
        ORDER BY _RawLog.col1, _RawLog.col2, _RawLog.col3, _RawLog.col4, _actionType, _RawLog.actiondate
    ) AS _FindChargeDate ON
        _FindChargeDate.col1 = T1.col1 AND _FindChargeDate.col2 = T1.col2 AND 
        _FindChargeDate.col3 = T1.col3 AND _FindChargeDate.col4 = T1.col4 AND 
        _FindChargeDate._actionType = T1._actionType AND
        _FindChargeDate._rowNumber = IF(T1.totalComms > 2, 3, IF(T1.totalComms > 1, 2, 1))
31.3.8.5. GROUP BY and HAVING
  • After WHERE before ORDER BY and HAVING
    • WHERE 1=1 GROUP BY col1 HAVING fieldDefinedinSELECT > 1 ORDER BY col1
  • GROUP BY col1 ORDER BY NULL
  • ORDER BY on non-grouped columns, see 31.3.8.4.1, the grouped query result on the left of the join, the order by table(s) is on the right of the join
31.3.8.6. COUNT
COUNT(*)
count all rows of a table
COUNT(columnA or expression)
count non-null only
(no term)
Use SUM(CASE WHEN 0 = 0 THEN 1 ELSE 0 END) or SUM(IF(aColumn = 1, 1, 0))~or ~SUM(IFNULL(aColumn, 0)) to achieve count if
31.3.8.7. REGEXP mysql:regex

Unless the field is binary, REGEXP is case-insensitive. 

WHERE aut_name REGEXP "^w"
WHERE aut_name REGEXP "^I am \"Great!\""
31.3.8.8. DELETE 
DELETE FROM t1 WHERE id = 123;

-- Delete records from t1 table only even though FROM is a join
DELETE t1 FROM t1
JOIN t2 ON t2.id = t1.id
WHERE t1.id = 123 AND t2.type = 'abc';

-- all matching rows of t1 and t2 are deleted
DELETE t1,t2 FROM t1
JOIN t2 ON t2.id = t1.id
WHERE t1.id = 123 AND t2.type = 'abc';
31.3.8.9. INSERT
  1. INSERT ... SELECT

    • If there's no record from SELECT, no row is inserted 

      INSERT INTO t1 (col1, col2)
SELECT t2.id, t3.name
FROM t2 JOIN t3 ON t2.t3bridge = t3.id
WHERE t2.score > 100;

      INSERT INTO timezones (id, timezonename)
SELECT 25, 'America/Phoenix'
WHERE NOT EXISTS (
    SELECT * FROM timezones WHERE id = 25
)

      INSERT INTO timezones (timezoneid, timezonename)
    (SELECT 25, 'America/Phoenix'
     FROM (SELECT 1 AS whocares) AS t
     WHERE NOT EXISTS (SELECT *
                       FROM timezones
                       WHERE timezoneid = 25))
UNION
(SELECT 27, 'America/Anchorage'
 FROM (SELECT 1 AS whocares) AS t
 WHERE NOT EXISTS (SELECT *
                   FROM timezones
                   WHERE timezoneid = 27))
UNION
(SELECT 29, 'America/Adak'
 FROM (SELECT 1 AS whocares) AS t
 WHERE NOT EXISTS (SELECT *
                   FROM timezones
                   WHERE timezoneid = 29))
UNION
(SELECT 31, 'Pacific/Honolulu'
 FROM (SELECT 1 AS whocares) AS t
 WHERE NOT EXISTS (SELECT *
                   FROM timezones
                   WHERE timezoneid = 31))

    • SELECT in brackets 

      INSERT INTO t1 (col1, col2)
VALUES
(
  1,
  (SELECT id FROM t2 WHERE t2_col1 = 1 LIMIT 1)
),
(
  2,
  (SELECT id FROM t2 WHERE t2_col1 = 2 LIMIT 1)
);
31.3.8.10. UPDATE
  1. Update fields in Multiple Tables
    SELECT ...
FROM {$from_statement}
WHERE {$where_statement}

-- eq. to UPDATE Statement
UPDATE
  {$from_statement}
SET {$set_statements}
WHERE {$where_statement}

    UPDATE node, node_revision
SET node.comment=1,
    node_revision.comment = 1
WHERE node_revision.nid = node.nid AND node.type <> 'blog'

    Update Joined Tables 

    UPDATE tableA a
JOIN tableB b ON a.a_id = b.a_id
JOIN tableC c ON b.b_id = c.b_id
SET b.val = a.val+c.val,
a.val2 = 'new'
WHERE a.val > 10 AND c.val > 10
  2. Update multiple rows in one query
    UPDATE table_users
SET cod_user = (
    case when user_role = 'student' then '1'
     when user_role = 'assistant' then '2'
     when user_role = 'admin' then '3'
    end),
date = 'set to this value for all matched rows'
WHERE user_role in ('student', 'assistant', 'admin') AND cod_office = '17389551';
  3. Can't use subquery in SET
    -- Set row #321's xml column to be the same as row #123
UPDATE a a1 join a a2 ON a2.id=123
SET a1.xml = a2.xml
where a1.id=321;
31.3.8.11. LIMIT, OFFSET
  • LIMIT {[offset,] row_count | row_count OFFSET offset}

  • Can't directly use LIMIT in IN sub query This doesn't work 

    SELECT *
FROM wp_posts
WHERE wp_posts.ID IN (
SELECT p2.ID FROM wp_posts p2 WHERE p2.post_type IN ('posts')
LIMIT 5)

    Do this 

    SELECT *
FROM wp_posts
WHERE wp_posts.ID IN (
SELECT * FROM (SELECT p2.ID FROM wp_posts p2 WHERE p2.post_type IN ('posts') LIMIT 5) AS sq1
)
  • if the outter query has LIMIT, then it will overwrite any subquery's LIMIT
31.3.8.12. EXISTS
  • If only checking existance in other tables but not returning any data from other tables, always use EXISTS or NOT EXISTS
  • The following returns nodes that have field_dealer = 101615 and field_ad_status = active but not returning any fields from tables field_data_*
SELECT count(*) FROM node AS n
WHERE n.type='ad_listing'
AND n.status = 1
AND EXISTS (
 SELECT *
 FROM field_data_field_dealer d
 INNER JOIN node nsub ON
   nsub.nid=d.entity_id
   AND d.bundle='ad_listing'
   AND d.field_dealer_target_id IN (101615)
 WHERE nsub.nid = n.nid
)
AND EXISTS (
 SELECT *
 FROM field_data_field_ad_status s
 INNER JOIN node nsub ON nsub.nid=s.entity_id AND s.bundle='ad_listing' AND s.field_ad_status_value = 'active'
 WHERE nsub.nid=n.nid
)
31.3.8.13. UNION 
select_statement_1
UNION [ALL]
select_statement_2
UNION [ALL]
select_statement_3
...
...
  • In a UNION query, there are at least two SELECT statements
  • The 2 SELECT statements must have the same number of columns and the the columns must have compatible data types
  • The column headings in each of the SELECT statements do not have to have the same name. The column headings in the result of a UNION query are always taken from the first SELECT statement
  • If you want to sort the result set of the UNION operation, you can only put an ORDER BY clause after the last SELECT statement. ORDER BY clause can't be specified in any other SELECT statements in the UNION query
  • The column(s) used in ORDER BY clause can only be taken from the first SELECT statement
  • If you don't specify an ORDER BY clause in the UNION query, the result set is always sorted by the first column
  • If you use UNION ALL, the entire result set from the second SELECT statement is appended to the first SELECT statement. In this case, there could be duplicate records in the unioned result set

  • If you only use UNION, MySQL removes duplicate rows from the final result set 

    SELECT s.id AS id, score.score, course.course, null AS certificate
FROM student s
  LEFT JOIN s_score score ON score.sid = s.id
  LEFT JOIN s_course course ON score.sid = s.id
UNION ALL
SELECT s.id AS id, null, null, cert.certificate
FROM student s
  LEFT JOIN s_certificate cert ON cert.sid = s.id
ORDER BY id
31.3.8.14. CASE, IF(expr1,expr2,expr3), IFNULL(expr1,expr2), NULLIF(expr1,expr2)
SELECT IF(1>2,2,3); -- 3
SELECT IF(1=1,2,3); -- 2
SELECT IF(STRCMP('1', '1') = 0, 2, 3); -- 2

-- IFNULL(expr1, expr2) :: return expr2 if expr1 is null
-- NULLIF(expr1, expr2) :: return NULL if expr1 = expr2

-- If value is NULL or another value, then convert and compare in one line
SELECT IF(field1 IS NULL OR field1 = '', 'empty', field1) = 'empty'
-- Or even shorter
SELECT IFNULL( NULLIF( field1, '' ), 'empty' ) = 'empty'

SELECT *
FROM t1
ORDER BY IF( colA = 'b', colB, NULL ) DESC, colC
LIMIT 1;

SELECT CASE 1
        WHEN 1 THEN 'one'
        WHEN 2 THEN 'two'
        ELSE 'more' END AS myColName;
-- one

SELECT CASE
    WHEN 1>0 THEN 'true'
    ELSE 'false' END;
-- 'true'

SELECT CASE BINARY 'B'
        WHEN 'a' THEN 1
        WHEN 'b' THEN 2 END;
-- NULL

SELECT *
FROM t1
ORDER BY
  CASE colA
       WHEN 'value1' THEN colB
       WHEN 'value2' THEN colC
       ELSE 'value3' END;

SELECT colA,
( SELECT COUNT(*) FROM t2) AS computedCol,
CASE computedCol -- Can't use computed column as CASE value!!
WHEN 2 then 1
END AS caseCol

-- Due to computed columns cannot be used, may need to repeat computed fields
SELECT colA,
(
SELECT CASE
WHEN COUNT(*) > 10 THEN 1
WHEN COUNT(*) > 0 THEN 2
END
FROM t2) AS caseCol

-- Nested CASE statements
SELECT CASE
   1 = 1 THEN CASE
     2 = 2 THEN 1 ELSE 2 END
31.3.8.15. DISTINCT
  • NULL is allowed
    Count non null Distinct values
    COUNT(DISTINCT my_expression)
  • SELECT DISTINCT col1, col2
  • Equivalent to GROUP BY
31.3.8.16. Math functions
  • All Math Functions
  • CONV( N, from_base, to_base )
    • Convert N from from_base to to_base, return a string
    • could be a number or a string
    • 2 and 36
    • Make sure the N and the returned number is less than 2^64, otherwise the result is not accurate
  • ROUND(), SUM()
    • e.g. ROUND(SUM(ca.assetvalue),2)
    • Both return as string in PHP!
  • Convert from MySQL string to numeric type in PHP
    intval($aColumnValue)
    convert to integer
    floatval($aColumnValue)

    convert to float 

    // no warning
var_dump(floatval('') == 0, floatval('  ') == 0, floatval(' 321.32 ') == 321.32, floatval('0.00') == 0, floatval(null) == 0)
    Never do $convertedValue = $aColumnValue + 0;
    throws warning when input value is empty string
    • null is fine and converted to 0 in int or float
31.3.8.17. Bitwise
SELECT
-- 1
CONV(131072, 10, 2) = 100000000000000000,
-- 1
1 << 17 = 131072
-- 1
LOG2(1 << 17) = 17,
-- 1
(131072 & (1 << 17)) = 131072,
-- 1
(131072 & (1 << 17)) > 0,
-- 1
CONV(1 << 17 | 1 << 1, 10, 2) = 100000000000000010,
-- 1, have the flag
((1 << 17) | (1 << 1)) & (1 << 1) > 0,
-- 1, have the flag
(1 << 17 | 1 << 1) & (1 << 1) = (1 << 1),
-- 1, doesn't have the flag
(1<<17 | 1<<1) & (1<<2) = 0,
-- 1, remove a flag
CONV(
((1 << 17) | (1 << 1)) & ~ (1 << 1),
10,2) = 100000000000000000,
-- 1, remove a flag
((1 << 17) | (1 << 1)) & ~ (1 << 1) = (1 << 17),
-- remove no flag
(((1 << 17) | (1 << 1)) & ~ 0) = ((1 << 17) | (1 << 1))
31.3.8.18. String functions 31.3.6.5
31.3.8.19. Comparison functions and Operators
  • COALESCE(value1, value2, ...)
    • Returns the first non-NULL value in the list, or NULL if there are no non-NULL values
    • WHERE COALESCE(col1, col2, col3) IS NOT NULL eq.
      • WHERE col1 IS NOT NULL OR col2 IS NOT NULL OR col3 IS NOT NULL
    • WHERE COALESCE(col1, col2, col3) IS NULL eq.
      • WHERE col1 IS NULL AND col2 IS NULL AND col3 IS NULL

  • Null-safe equal to operator 

    -- Like `=` operator. Return 1 if both operands are NULL. Return 0 if one operand is NULL
SELECT 1 <=> 1, NULL <=> NULL, 1 <=> NULL;
-- 1, 1, 0
SELECT 1 = 1, NULL = NULL, 1 = NULL;
-- 1, NULL, NULL
31.3.8.20. Transaction
SET autocommit = 0;
-- SET autocommit = OFF;

START TRANSACTION;

SELECT * FROM abc

COMMIT;

SET autocommit = 1;
31.3.8.21. Rows to columns
/*
Return one row
ABC XYZ
123 321

Instead of returning 2 rows
ABC 123
XYZ 321
*/

SELECT (
       SELECT t1.templateid
       FROM templates t1
       WHERE t1.companyid = ?
         AND t1.templatekey = 'ABC'
       LIMIT 1
   ) AS ABC,
   (
       SELECT t2.templateid
       FROM templates t2
       WHERE tt2.companyid = ?
         AND tt2.templatekey = 'XYZ'
       LIMIT 1
   ) AS SMS

31.3.9. Admin queries

31.3.9.1. CREATE TABLE IF NOT EXITS
31.3.9.2. ALTER TABLE
-- MODIFY eq. MODIFY COLUMN can change definition only
MODIFY [COLUMN] col_name column_definition
        [FIRST | AFTER col_name]

column_definition: {
    data_type [NOT NULL | NULL] [DEFAULT default_value]
      [AUTO_INCREMENT] [UNIQUE [KEY]] [[PRIMARY] KEY]
      [COMMENT 'string']
      [COLLATE collation_name]
      [COLUMN_FORMAT {FIXED | DYNAMIC | DEFAULT}]
      [STORAGE {DISK | MEMORY}]
      [reference_definition]
  | data_type
      [COLLATE collation_name]
      [GENERATED ALWAYS] AS (expr)
      [VIRTUAL | STORED] [NOT NULL | NULL]
      [UNIQUE [KEY]] [[PRIMARY] KEY]
      [COMMENT 'string']
      [reference_definition]
}

ALTER TABLE `t1` MODIFY `col1` VARCHAR(255);
ALTER TABLE `t1` MODIFY `col1` VARCHAR(255) FIRST; -- AFTER can be used

-- Duplicate a column
ALTER TABLE `t1` ADD `col1duplicate` VARCHAR(255);
UPDATE `t1` SET `col1duplicate` = `col1`;

-- CHANGE eq. CHANGE COLUMN can change column name
-- The order ~FIRST | AFTER anotherColumn~ is optional
-- To get column definition, refer to mysql:show create table
ALTER TABLE `t1` CHANGE `col1` `col1newname` VARCHAR(255) FIRST;

ALTER TABLE `t1` CHANGE COLUMN `col1` `col1` varchar(255) default 'a';

ALTER TABLE `t1` ADD `col1` tinyint DEFAULT 0; -- set a default value
ALTER TABLE `t1` ADD `col1` longtext NULL; --  can be null

ALTER TABLE `t1` ADD `col1` VARCHAR(255) NULL AFTER `col2`;

-- RENAME
ALTER TABLE `t1` RENAME COLUMN original_column_name TO new_column_name; -- v8.0+

-- ADD eq. ADD COLUMN
ALTER TABLE `t1` ADD COLUMN `col1` VARCHAR(10) DEFAULT NULL; -- default null also means can be null
ALTER TABLE `t1` ADD COLUMN `col1` VARCHAR(10) DEFAULT NULL,
  ADD COLUMN `col2` bigint(20) unsigned DEFAULT NULL,
  ADD COLUMN `col3` tinyint(1) unsigned DEFAULT 0; -- one query multiple alter's

-- DROP column
ALTER TABLE `t1` DROP COLUMN `col1`;
31.3.9.3. Duplicate a table
-- to copy indexes and triggers
CREATE TABLE new_table LIKE old_table;
INSERT INTO new_table SELECT * FROM old_table;
31.3.9.4. Table Status, Table Settings, Table Definition

  • Filter tables 

    -- any table with suffix `x`
SHOW TABLES LIKE '%x';
  1. Show table settings for all tables
    SHOW TABLE STATUS FROM mydatabase;

-- filter by table name
SHOW TABLE STATUS WHERE Name LIKE '%users%';
    Engine
    Storage Engine
    • MyISAM
      • can roll back if a transaction fails
    Version
    e.g. 10
    Rows
    number of records. MyISAM shows exact while InnoDB shows approximation may vary by as much as 40% to 50%
    Avg_row_length
    avg row size. Refer to mysql:row size
    Collation
    utf8mb4_general_ci
  2. DDL - Data Definition Language, Table definition

    • Column definition and index keys 

      -- Column definition and index keys
SHOW CREATE TABLE myTable\G;
    • 31.3.9.8
31.3.9.5. Routines
SELECT * FROM INFORMATION_SCHEMA.ROUTINES
31.3.9.6. Deadlock
TS
Lock wait timeout exceeded; try restarting transaction
-- default lock wait is 50 secs
show variables like 'innodb_lock_wait_timeout';

-- see what is locked
SHOW ENGINE INNODB STATUS\G;

-- Sample output :: https://gist.github.com/levonlee/b816cf8f06fb2d23946661740d356d91

MySQL thread id 19, OS thread handle 140471672895232, query id 209779 localhost 127.0.0.1 root update
INSERT INTO yourQuery

SHOW PROCESSLIST;
-- eq. to 
-- SELECT * FROM information_schema.processlist ORDER BY id;
-- id is the MySQL thread id e.g. 19

-- May see more info about that thread
SELECT * FROM performance_schema.events_statements_history WHERE thread_id = (SELECT THREAD_ID FROM performance_schema.threads WHERE PROCESSLIST_ID = 19)

-- kill this thread meaning to kill the whole connection
KILL CONNECTION 19;
-- eq. to
-- KILL 19;

-- To check if it's killed, run SHOW PROCESSLIST;
31.3.9.7. Transpose query results
mysql -u username -p
select * from atable\G
31.3.9.8. Table columns
To get index keys in a table
31.3.9.4.2
  1. INFORMATION_SCHEMA.COLUMNS

    • TABLE_NAME Search tables by column name 

      SELECT DISTINCT TABLE_NAME 
FROM INFORMATION_SCHEMA.COLUMNS
WHERE COLUMN_NAME IN ('columnA','ColumnB') AND TABLE_SCHEMA='YourDatabase';

    • COLUMN_NAME All columns of a table 

      SELECT column_name, COLUMN_TYPE, COLUMN_DEFAULT, IS_NULLABLE, COLUMN_COMMENT
FROM information_schema.columns
WHERE table_schema = DATABASE() -- current database
AND table_name = 'mytablename'
    • e.g. decimal(10,2) unsigned
    • e.g. bigint
    • e.g. auto_increment
    • e.g. 0, 'default string'
    • YES or NO
    • COLUMN_COMMENT

    • See ALTER TABLE 

      alter table myTable
    modify colA tinyint unsigned default 0 null;

ALTER TABLE myTable
  MODIFY colA {COLUMN_TYPE} DEFAULT {COLUMN_DEFAULT} {IS_NULLABLE}
  2. Column definition SHOW FULL COLUMNS

    • Collation, privileges, comments 

      SHOW FULL COLUMNS FROM myTable;
SHOW FULL COLUMNS FROM myTable WHERE Field = 'col1';

-- only show Type, Null, Key Default, Extra. They're all the same
DESCRIBE myTable;
EXPLAIN myTable;
31.3.9.9. EXPLAIN
  • Use it before SELECT, DELETE, INSERT, REPLACE AND UPDATE statements
  • EXPLAIN format=JSON SELECT * FROM aTable
  • See SHOW FULL PROCESSLIST
  1. JSON
    • query_block
      • cost_info
        • query_cost
      • ordering_operation
        nested_loop
        array
        • The sequence matters
        • array item
          • table
            using_index
            if true, great! True means all the columns needed from the table are found in the INDEX being used.
            • False or not exists otherwise
            • Same for GROUP BY
            access_type
            join type. From the best to the worst type
            system
            the table has only one row
            const

            at most one matching row. It's used when you compare all parts of a PRIMARY KEY or UNIQUE index and they are constant values. 

            SELECT * FROM tbl_name
  WHERE primary_key_part1=1 AND primary_key_part2=2;
            eq_ref

            one row is read from this table for each combination of rows from the previous tables. It's used when all parts of an index are used by the join and the index is a PRIMARY KEY or UNIQUE NOT NULL index. Operator = 

            SELECT * FROM ref_table,other_table
  WHERE ref_table.key_column = other_table.column;

SELECT * FROM ref_table,other_table
  WHERE ref_table.key_column_part1 = other_table.column
  AND ref_table.key_column_part2 = 1;
            ref
            all rows with matching index values are read from this table for each combo of rows from the previous tables.

            • Examples 

              SELECT * FROM ref_table WHERE key_column= expr;

SELECT * FROM ref_table,other_table
  WHERE ref_table.key_column = other_table.column;

SELECT * FROM ref_table,other_table
  WHERE ref_table.key_column_part1 = other_table.column
  AND ref_table.key_column_part2 = 1;
            • It's used if the join uses only a leftmost prefix of the key or if the key is not a PRIMARY KEY or UNIQUE index (in other words, if the join cannot select a single row based on the key value). If the used key that matches only a few rows, this is a good join type.
              • Operator: = or <=>
            fulltext
            join using FULLTEXT index
            (no term)
            ref_or_null
            (no term)
            index_merge
            (no term)
            unique_subquery
            (no term)
            index_subquery
            (no term)
            range
            (no term)
            index
            ALL
            a full table scan is done
            (no term)
            filtered
            100
            bad. Means no filtering of rows occured
            (no term)
            key
            • The index that is actually used
            • no index is used
            (no term)
            cost_info
            (no term)
            materialized_from_subquery
            • query_block
              • cost_info
              • grouping_operation
                • table
  2. Non JSON
    Output
    https://dev.mysql.com/doc/refman/8.0/en/explain-output.html
    type
    same as JSON access_type
    operator
    =
    (no term)
    key
    null
    when it's NULL means it's a full table scan
    • Or the target index key is not used due to the provided value (from previous tables) does not exist in this table. Extra column may show Impossible ON condition
    rows
    number of rows to scan
    • when key is NULL, it'll be the table's total number of rows
31.3.9.10. SHOW FULL PROCESSLIST
31.3.9.11. SIGNAL / RESIGNAL throw error or warning
  1. Basics
    SIGNAL condition_value
    [SET signal_information_item
    [, signal_information_item] ...]

condition_value: {
    SQLSTATE [VALUE] sqlstate_value
  | condition_name
}

signal_information_item:
    condition_information_item_name = simple_value_specification

condition_information_item_name: {
    CLASS_ORIGIN
  | SUBCLASS_ORIGIN
  | MESSAGE_TEXT
  | MYSQL_ERRNO
  | CONSTRAINT_CATALOG
  | CONSTRAINT_SCHEMA
  | CONSTRAINT_NAME
  | CATALOG_NAME
  | SCHEMA_NAME
  | TABLE_NAME
  | COLUMN_NAME
  | CURSOR_NAME
}

    SIGNAL
SQLSTATE '01000'
SET MESSAGE_TEXT = 'my warning';

    CREATE PROCEDURE p (pval INT)
BEGIN
  DECLARE specialty CONDITION FOR SQLSTATE '45000';
  IF pval = 0 THEN
    SIGNAL SQLSTATE '01000';
    -- '01' :: warning class, which does not terminate
  ELSEIF pval = 1 THEN
    SIGNAL SQLSTATE '45000'
  SET MESSAGE_TEXT = 'An error occurred';
  ELSEIF pval = 2 THEN
    SIGNAL specialty
  SET MESSAGE_TEXT = 'An error occurred';
  ELSE
    SIGNAL SQLSTATE '01000'
  SET MESSAGE_TEXT = 'A warning occurred', MYSQL_ERRNO = 1000;
    -- The last SIGNAL replaces MESSAGE_TEXT AND MYSQL_ERRNO set previously
    SIGNAL SQLSTATE '45000'
  SET MESSAGE_TEXT = 'An error occurred', MYSQL_ERRNO = 1001;
  END IF;
END;
  2. SQLSTATE
    • 5-character string
      • First 2 characters are called class
      • generic value, exception, means "unhandled user-defined exception"
    • Overrides MYSQL_ERRNO
      • Classes
        '00'
        success. Do not use this in SIGNAL / RESIGNAL
        '01'
        warning but not failing the statement
        '02'
        not found. By default, it will work just as warning: not failing the statement
        > '02'
        exception, fails the statement
  3. RESIGNAL
    • Pass upper level warning/error onto the current level
    • Must be inside a 31.3.9.12
    DELIMITER $$

CREATE PROCEDURE Divide(IN numerator INT, IN denominator INT, OUT result double)
BEGIN
    DECLARE division_by_zero CONDITION FOR SQLSTATE '22012';

    DECLARE CONTINUE HANDLER FOR division_by_zero 
    RESIGNAL SET MESSAGE_TEXT = 'Division by zero / Denominator cannot be zero';
    -- 
    IF denominator = 0 THEN
        SIGNAL division_by_zero;
    ELSE
        SET result := numerator / denominator;
    END IF;
END
31.3.9.12. Handler
  • https://dev.mysql.com/doc/refman/8.0/en/declare-handler.html
  • Handler declarations must appear after variable or condition declarations
  • condition_value
    SQLWARNING
    Shorthand for the class of SQLSTATE values that begin with '01'
    NOT FOUND
    Shorthand for the class of SQLSTATE values that begin with '02'
    SQLEXCEPTION
    Shorthand for the class of SQLSTATE values that do not begin with '00', '01', or '02'
DECLARE handler_action HANDLER
    FOR condition_value [, condition_value] ...
    statement

handler_action: {
    CONTINUE
  | EXIT
  | UNDO
}

condition_value: {
    mysql_error_code
  | SQLSTATE [VALUE] sqlstate_value
  | condition_name
  | SQLWARNING
  | NOT FOUND
  | SQLEXCEPTION
}
31.3.9.13. Profiling
SET profiling = 1;
-- Run query #1
-- Run query #2
SHOW PROFILES;
-- Query_ID and Duration
31.3.9.14. Row size and number of columns

31.3.10. User-Defined Variables

SET @s3 = 'http://s3.amazon.com/abc/';
-- eq. to
SET @s3 := 'http://s3.amazon.com/abc/';

SET @s1 = 'da', @s2 = 'ba';

-- variable name is alphanumeric with `.`, `_` and `$`
-- @'my-var', @"my-var", @`my-var`
-- Case-insensitive, up to 64 characters

-- integer, decimal, floating-point, binary or nonbinary string, or NULL

SELECT @s3;

31.3.11. Routines, Procedures, User Defined Function

  • Stored Procedure and Stored Functions are stored under the schema as routines
    • function returns a value
  • db_name.sp_name

  • List all routines 

    SELECT * FROM INFORMATION_SCHEMA.ROUTINES
WHERE 0=0
-- AND ROUTINE_TYPE = 'FUNCTION';
-- AND ROUTINE_TYPE = 'PROCEDURE';
-- AND ROUTINE_NAME = 'myname';
31.3.11.1. Procedure 
CREATE
    [DEFINER = user]
    PROCEDURE [IF NOT EXISTS] sp_name ([proc_parameter[,...]])
    [characteristic ...] routine_body
  • proc_parameter
    • Types
      IN
      the procedure might modify but the caller cannot see the modification
      OUT
      the value is passed back from the procedure back to the caller. Initial value is NULL
      (no term)
      INOUT
DELIMITER //

CREATE PROCEDURE GetAllProducts()
BEGIN
    SELECT *  FROM products;
END //

DELIMITER ;

-- CALL GetAllProducts();

DELIMITER //

CREATE PROCEDURE GetAllProducts(IN country CHAR(3), OUT cities INT)
BEGIN
    SELECT COUNT(*) INTO cities  FROM products
    WHERE CountryCode = country;;
END //

DELIMITER ;

-- CALL GetAllProducts('JPN', @cities);
-- SELECT @cities;
31.3.11.2. Characteristics 
characteristic:
    COMMENT 'string'
  | LANGUAGE SQL
  | [NOT] DETERMINISTIC
  | { CONTAINS SQL | NO SQL | READS SQL DATA | MODIFIES SQL DATA }
  | SQL SECURITY { DEFINER | INVOKER }

Don't specify LANGUAGE SQL

A routine is considered “deterministic” if it always produces the same result for the same input parameters, and “not deterministic” otherwise. If neither DETERMINISTIC nor NOT DETERMINISTIC is given in the routine definition, the default is NOT DETERMINISTIC. To declare that a function is deterministic, you must specify DETERMINISTIC explicitly.

Non-deterministic examples are data modification, have UPDATE, INSERT or DELETE statement(s). Default without SET GLOBAL log_bin_trust_function_creators = 1; is NOT DETERMINISTIC Without specifying DETERMINISTIC and with the default setting, this error throws: 

This function has none of DETERMINISTIC, NO SQL, or READS SQL DATA in its declaration and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable)

{ CONTAINS SQL | NO SQL | READS SQL DATA | MODIFIES SQL DATA }

  • CONTAINS SQL Default, indicates that the routine does not contain statements that read or write data. Examples of such statements are SET @x = 1 or DO RELEASE_LOCK('abc'), which execute but neither read nor write data.
  • indicates that the routine contains no SQL statements.
  • DATA indicates that the routine contains statements that read data (for example, SELECT), but not statements that write data.
  • indicates that the routine contains statements that may write data (for example, INSERT or DELETE).

The parameter list enclosed within parentheses must always be present. If there are no parameters, an empty parameter list of () should be used. Parameter names are not case sensitive.

31.3.11.3. Update definer
update mysql.proc set definer='root@localhost' where name = 'myprocedurename';

-- For MySQL v8.0, there's no mysql.proc table
31.3.11.4. Stored Procedure
  1. Syntax
    CREATE
    [DEFINER = { user | CURRENT_USER }]
    PROCEDURE sp_name ([proc_parameter[,...]])
    [characteristic ...] routine_body

proc_parameter:
    [ IN | OUT | INOUT ] param_name type
    IN
    Default. It passes a value into a procedure. The procedure might modify the value, but the modification is not visible to the caller when the procedure returns.
    OUT
    passes a value from the procedure back to the caller. Its initial value is NULL within the procedure, and its value is visible to the caller when the procedure returns. An INOUT parameter is initialized by the caller, can be modified by the procedure, and any change made by the procedure is visible to the caller when the procedure returns.
    (no term)
    Use () if there is no parameter
    (no term)
    In body, stored function can be used

    For each OUT or INOUT parameter, pass a user-defined variable in the CALL statement that invokes the procedure so that you can obtain its value when the procedure returns. If you are calling the procedure from within another stored procedure or function, you can also pass a routine parameter or local routine variable as an IN or INOUT parameter.

    Routine parameters cannot be referenced in statements prepared within the routine

    Each SELECT statement that does not insert into a table or a variable will produce a result set. 

    mysql> delimiter //

mysql> CREATE PROCEDURE simpleproc (OUT param1 INT)
    -> BEGIN
    ->   SELECT COUNT(*) INTO param1 FROM t;
    -> END; //
Query OK, 0 rows affected (0.00 sec)

mysql> delimiter ;

mysql> CALL simpleproc(@a);
Query OK, 0 rows affected (0.00 sec)

mysql> SELECT @a;
+------+
| @a   |
+------+
| 3    |
+------+
1 row in set (0.00 sec)
  2. Prepare Statement, Convert delimited string into temp table

    A prepare statement is unique to a session. If it's not deallocated, it will be deallocated automatically when the session ends. mysql:string:list:temp table 

    DROP PROCEDURE IF EXISTS `getArticlesByCats`;
CREATE PROCEDURE getArticlesByCats(IN pDelim VARCHAR(32), IN pStr TEXT )
DETERMINISTIC
MODIFIES SQL DATA
  BEGIN
    DROP TABLE IF EXISTS li_temp_explode;
    CREATE TEMPORARY TABLE li_temp_explode (id INT AUTO_INCREMENT PRIMARY KEY NOT NULL, word VARCHAR(40));
    SET @sql := CONCAT('INSERT INTO li_temp_explode (word) VALUES (', REPLACE(QUOTE(pStr), pDelim, '\'), (\''), ')');
    PREPARE myStmt FROM @sql;
    EXECUTE myStmt;
    DEALLOCATE PREPARE myStmt;
    -- Later use the temp table in IN statement
    SELECT *
    FROM node n
    WHERE EXISTS(
  SELECT * 
  FROM cats c
  WHERE c.cid IN (
    SELECT CAST(word AS UNSIGNED)
    FROM li_temp_explode
  )
    )
  END;

    word column is string 

    CALL `getArticlesByCats`(',', '1,31,5,6,7,8,9,10');
SELECT * FROM li_temp_explode;
    List all stored procedures of all dbs that the user has access to
    SHOW PROCEDURE STATUS;
    List stored procedures of a db
    SHOW PROCEDURE STATUS WHERE db = 'mydbname'
    List stored procedures by name
    SHOW PROCEDURE STATUS WHERE name LIKE '%product%'
    Show the code of a stored procedure
    SHOW PROCEDURE my_stored_procedure_name
    Drop a stored procedure
    DROP PROCEDURE IF EXISTS my_sp_name
31.3.11.5. User Defined Function mysql:udf
  • UDF always returns a value
  1. Syntax
    • Refer to [[mysql:routine:characteristics][mysql:routine:characteristics
    CREATE
    [DEFINER = { user | CURRENT_USER }]
    FUNCTION sp_name ([func_parameter[,...]])
    RETURNS type
    [characteristic ...] routine_body

func_parameter:
    param_name type

type:
    Any valid MySQL data type

    SET @lis3 = 'http://s3.amazonaws.com/abc/';
SET @lifs = 'http://abc.com/sites/default/files/';
SET @litt = 'http://abc.com/';
SET @lir1 = '^(s3://)(.*)';
SET @lir2 = CONCAT(@lis3, '\\2');
SET @lir3 = '^(public://)(.*)';
SET @lir4 = CONCAT(@lifs, '\\2');

DROP FUNCTION IF EXISTS `liFileUri`;

CREATE FUNCTION `liFileUri` (`uri` VARCHAR(255)) 
  RETURNS VARCHAR(255)
DETERMINISTIC
NO SQL
  BEGIN
    RETURN REGEXP_REPLACE(REGEXP_REPLACE(uri, @lir1, @lir2), @lir3, @lir4);
  END;

DROP FUNCTION IF EXISTS `liConCat2`;
CREATE FUNCTION `liConCat2`(`v1` TEXT, `v2` TEXT)
  RETURNS TEXT
DETERMINISTIC
NO SQL
  BEGIN
    DECLARE s TEXT;
    IF (v1 IS NULL AND v2 IS NULL)
    THEN SET s = NULL;
    ELSE SET s = CONCAT_WS(0x1F, v1, v2);
    END IF;
    RETURN s;
  END;

-- UDF can't have optional parameter. e.g. have to create liConCat3 to concat 3 fields

DROP FUNCTION IF EXISTS `liFixNull`;
CREATE FUNCTION `liFixNull`(`text` TEXT)
  RETURNS TEXT
DETERMINISTIC
  BEGIN
-- Use this with CONCAT('aString', liFixNull(aColumn)) is useful because CONCAT returns NULL if any is null
    RETURN CASE text
       WHEN 'NULL' THEN NULL
       WHEN '' THEN NULL
       ELSE text
       END;
  END;
  2. Return a single value from SELECT statement
    DROP FUNCTION IF EXISTS udfreturnselect;

DELIMITER ;;

CREATE FUNCTION udfreturnselect(parentQueryT1Col1_ BIGINT(20),
                parentQueryT2Col1_ DECIMAL(20, 2))
    RETURNS INT(11)
    DETERMINISTIC
    CONTAINS SQL

BEGIN
    DECLARE udfreturnselect_result INT;

    SELECT CASE
       WHEN (IFNULL(parentQueryT2Col1_, 0) - SUM(IFNULL(t3.invamount, 0)) < 0.01 AND
             parentQueryT2Col1_ > 0) THEN 1
       WHEN (IFNULL(parentQueryT2Col1_, 0) - SUM(IFNULL(t3.invamount, 0)) > 0 AND
             SUM(IFNULL(t3.invamount, 0)) > 0) THEN 2
       ELSE 3
       END
    INTO udfreturnselect_result
    FROM t3
    WHERE t3.roundid = parentQueryT1Col1_
  AND t3.replaced = 0;

    RETURN udfreturnselect_result;
END ;;

DELIMITER ;

31.3.12. SQL Fiddle

Schema 

CREATE TABLE student (
`sid` INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY, 
`firstname` VARCHAR(20) NOT NULL,
`lastname` VARCHAR(20) NOT NULL,
`reg_date` TIMESTAMP
);

INSERT INTO student ( firstname, lastname, reg_date )
VALUES 
( 'first1', 'last1', current_timestamp), -- 1
( 'first2', 'last2', current_timestamp), -- 2
( 'first3', 'last3', current_timestamp); -- 3

CREATE TABLE course (
`cid` INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY, 
`coursename` VARCHAR(20) NOT NULL,
`semaster_date` BIGINT(20) UNSIGNED NOT NULL
);

INSERT INTO course ( coursename, semaster_date )
VALUES 
( 'Math',    0 ), -- 1
( 'English', 0 ), -- 2
( 'PHP',     1593015426 ); -- 3

CREATE TABLE s_course (
`scid` INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY, 
`sid` INT(6) UNSIGNED NOT NULL,
`cid` INT(6) UNSIGNED NOT NULL
);

INSERT INTO s_course ( sid, cid )
VALUES 
( 1, 1 ), -- 1
( 1, 2 ), -- 2
( 2, 3 ); -- 3

CREATE TABLE s_score (
`ssid` INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY, 
`scid` INT(6) UNSIGNED NOT NULL,
`score` INT(3) UNSIGNED NOT NULL,
`midterm_score` INT(3) UNSIGNED NOT NULL DEFAULT 0,
`assignment_score` INT(3) UNSIGNED NOT NULL DEFAULT 0,
`final_exam_score` INT(3) UNSIGNED NULL DEFAULT 0,
`test_date` BIGINT(20) UNSIGNED NOT NULL
);

INSERT INTO s_score ( scid, score, midterm_score, assignment_score, final_exam_score, test_date )
VALUES 
( 1, 20, 21, 22, 23, 0 ), -- Student 1
( 2, 30, 31, 32, 33, 0 ), -- 2
( 3, 40, 41, 42, 43, 0 ), -- 3
( 3, 40, 41, 42, null, 0 ), -- 3
( 1, 50, 51, 52, 53, 1593015426 ), -- 1
( 1, 60, 61, 62, 63, 1594015426 ), -- 1
( 2, 70, 71, 72, 73, 1595015426 ), -- 2
( 1, 60, 61, 62, 63, 1594015426 ); -- 1

31.3.13. MySQLTutorial.org

31.3.14. Dump, Restore 

mysqldump -u user -p database_name > database_name_backup.sql

# `--single-transaction` prevents table from being locked
mysqldump --single-transaction --quick --max_allowed_packet=512M --compress --routines --triggers --events -uuser -p database_name > backup.sql

# Assuming there's enough disk space
mysqldump -uuser -p database_name > backup.sql && gzip backup.sql

# Remove encryption
sed -i "s/ENCRYPTION='Y'//g" backup.sql

# Not enough disk space then use pipe
mysqldump -uroot -p database_name | gzip > backup.sql.gz

# dump specific tables
mysqldump -uroot -pMyPassword --single-transaction --quick --compress my_database t1 t2 t3 | gzip > a.sql.gz
# No `CREATE DATABASE`

# don't dump some tables
mysqldump -uroot -p --ignore-table=database_name.tableNameToIgnore database_name

# Use config file to avoid typing password
mysqldump --defaults-extra-file=/path/to/.mydb.cnf -u user db_name > db_name_bk.sql

# without any -u or -p, the user config file `~/.my.cnf` is used. Refer to mariadb:config
mysqldump db_name > db_name.sql

# dump a table
mysqldump db_name table_name > table_name.sql

# restore one or more tables
mysql -u username -p db_name < /path/to/table_name.sql

zcat 1.sql.gz | mysql -uroot -p my_db

# Change a table name in mysqldump file
# /*!40000 ALTER TABLE `my_table` ENABLE KEYS */;
(_oldname="my_table"; _newname="my_table_new"; cat 1.sql \
| sed "s/^CREATE TABLE \`${_oldname}\`/CREATE TABLE \`${_newname}\`/g" \
| sed "s/^DROP TABLE IF EXISTS \`${_oldname}\`/DROP TABLE IF EXISTS \`${_newname}\`/g" \
| sed "s/^INSERT INTO \`${_oldname}\`/INSERT INTO \`${_newname}\`/g" \
| sed "s/^-- Table structure for table \`${_oldname}\`/-- Table structure for table \`${_newname}\`/g" \
| sed "s/^-- Dumping data for table \`${_oldname}\`/-- Dumping data for table \`${_newname}\`/g" \
| sed "s/^LOCK TABLES \`${_oldname}\`/LOCK TABLES \`${_newname}\`/g" \
| sed "s/^\/\*\!40000 ALTER TABLE \`${_oldname}\`/\/\*\!40000 ALTER TABLE \`${_newname}\`/g" \
| grep --color "${_oldname}")

(_oldname="my_table"; _newname="my_table_new"; gunzip < 1.sql.gz \
| sed "s/^CREATE TABLE \`${_oldname}\`/CREATE TABLE \`${_newname}\`/g" \
| sed "s/^DROP TABLE IF EXISTS \`${_oldname}\`/DROP TABLE IF EXISTS \`${_newname}\`/g" \
| sed "s/^INSERT INTO \`${_oldname}\`/INSERT INTO \`${_newname}\`/g" \
| sed "s/^-- Table structure for table \`${_oldname}\`/-- Table structure for table \`${_newname}\`/g" \
| sed "s/^-- Dumping data for table \`${_oldname}\`/-- Dumping data for table \`${_newname}\`/g" \
| sed "s/^LOCK TABLES \`${_oldname}\`/LOCK TABLES \`${_newname}\`/g" \
| sed "s/^\/\*\!40000 ALTER TABLE \`${_oldname}\`/\/\*\!40000 ALTER TABLE \`${_newname}\`/g" \
| mysql -uroot -p my_db)

# Change a table name in mysqldump file

# /*!40000 ALTER TABLE `my_table` ENABLE KEYS */;
(_oldname="my_table"; _newname="my_table_new"; cat 1.sql \
| sed "s/^CREATE TABLE \`${_oldname}\`/CREATE TABLE \`${_newname}\`/g" \
| sed "s/^DROP TABLE IF EXISTS \`${_oldname}\`/DROP TABLE IF EXISTS \`${_newname}\`/g" \
| sed "s/^INSERT INTO \`${_oldname}\`/INSERT INTO \`${_newname}\`/g" \
| sed "s/^-- Table structure for table \`${_oldname}\`/-- Table structure for table \`${_newname}\`/g" \
| sed "s/^-- Dumping data for table \`${_oldname}\`/-- Dumping data for table \`${_newname}\`/g" \
| sed "s/^LOCK TABLES \`${_oldname}\`/LOCK TABLES \`${_newname}\`/g" \
| sed "s/^\/\*\!40000 ALTER TABLE \`${_oldname}\`/\/\*\!40000 ALTER TABLE \`${_newname}\`/g" \
| grep --color "${_oldname}")

.mydb.cnf 

[mysqldump]
password='enclosedwithdoublequotes'

MySQL is inside a container some-mysql and dump it on host 

docker exec some-mysql sh -c 'exec mysqldump --all-databases -uroot -p"your-passowrd"' > /some/path/on/your/host/all-databases.sql

Restore 

cat backup.sql | docker exec -i CONTAINER /usr/bin/mysql -u root --password=root DATABASE

Dump a remote database 

mysqldump -P3306 -h192.168.20.151 -u root -p databasename > c:/my.sql
31.3.14.1. Dump data
-- MySQL dump 10.13  Distrib 5.7.37, for Linux (x86_64)
--
-- Host: localhost    Database: loanstudio
-- ------------------------------------------------------
-- Server version   5.7.37

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;


--
-- Table structure for table `abc`
--

DROP TABLE IF EXISTS `abc`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `abc` (
-- some columns
) ENGINE=InnoDB AUTO_INCREMENT=341 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Dumping data for table `lawyerintegrationbills`
--

LOCK TABLES `abc` WRITE;
/*!40000 ALTER TABLE `abc` DISABLE KEYS */;
INSERT INTO `abc` VALUES (
-- row
)
-- rows
-- ,()
;
/*!40000 ALTER TABLE `abc` ENABLE KEYS */;
UNLOCK TABLES;

-- repeat for tables

--
-- Dumping events for database 'mydb'
--

/*!50003 SET sql_mode              = @saved_sql_mode */ ;
/*!50003 SET character_set_client  = @saved_cs_client */ ;
/*!50003 SET character_set_results = @saved_cs_results */ ;
/*!50003 SET collation_connection  = @saved_col_connection */ ;
/*!50003 DROP FUNCTION IF EXISTS `myfuncs` */;
/*!50003 SET @saved_cs_client      = @@character_set_client */ ;
/*!50003 SET @saved_cs_results     = @@character_set_results */ ;
/*!50003 SET @saved_col_connection = @@collation_connection */ ;
/*!50003 SET character_set_client  = latin1 */ ;
/*!50003 SET character_set_results = latin1 */ ;
/*!50003 SET collation_connection  = latin1_swedish_ci */ ;
/*!50003 SET @saved_sql_mode       = @@sql_mode */ ;
/*!50003 SET sql_mode              = 'NO_ENGINE_SUBSTITUTION' */ ;
DELIMITER ;;
CREATE DEFINER=`root`@`localhost` FUNCTION `myfunceffstatus`(date_ bigint, status_ int, intadj_ bigint, dischargedate_ bigint, effdate_ bigint) RETURNS int(11)
    DETERMINISTIC
begin


if status_=0 or status_=3 then return status_; end if;

if status_=1 then
    if date_<intadj_ then return 0; end if;
    if date_>effdate_ then return 4; end if;
    return 1;
end if;

if status_=2 then
    if date_ <intadj_ then return 0; end if;
    if date_<dischargedate_ then return 1; end if;
    return 2;
end if;

end ;;
DELIMITER ;

-- repeat for routines/functions

/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;

/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;

-- Dump completed on 2022-04-15 10:53:06
31.3.14.2. Dump table definition only
# all tables in a db
mysqldump -uroot -p --no-data mydatabase > my-all-tables-def.sql
# for a table only
mysqldump -uroot -p --no-data mydatabase mytable1 > mytable1-def.sql
# --no-create-info dump data not definition
31.3.14.3. Dump query result as CSV
  1. SELECT ... INTO OUTFILE
    • It's for the purpose of LOAD DATA. It has its limitation
      • Does not work for field value that is multiple lines
      • It requires special permissions otherwise it will throw this error
        • ERROR 1045 (28000) at line 1: Access denied for user 'root'@'%' (using password: YES)
    SELECT *
FROM a
INTO OUTFILE '/var/lib/mysql-files/a.txt'
-- default ESCAPED BY is '\', disable escape: ESCAPED BY ''
FIELDS TERMINATED BY ',' ENCLOSED BY '"' 
-- place ESCAPED BY here before LINES 
LINES TERMINATED BY '\n'

    mysql -e "SELECT * FROM a INTO OUTFILE '/var/lib/mysql-files/a.txt' FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\\n';" -u root -p abc

    -- need to use this directory
SHOW VARIABLES LIKE "secure_file_priv";
  2. mysql -e "SELECT * FROM t1 ORDER BY id DESC LIMIT 1\G" -uroot -pABC databasename > /tmp/lili.tsv
    • Dump as tsv (tab delimited)

    • Script file can be used instead of -e 

      mysql -uroot -pABC databasename < script.sql > /tmp/lili.tsv
31.3.14.4. Dump a row as INSERT statements
# --no-create-info, -t :: no CREATE TABLE statements
# --where='where_condition', -w 'where_condition'
# --complete-insert :: add field names in the INSERT statement
# --compact :: remove comments. This enables --skip-add-drop-table, --skip-add-locks, --skip-comments, --skip-disable-keys, and --skip-set-charset
# --extended-insert, -e :: INSERT statements using multiple-row syntax that includes several VALUES lists
# --set-gtid-purged=OFF :: just to make sure only INSERT statements are returned
mysqldump myDB myTable --no-create-info --compact --complete-insert --set-gtid-purged=OFF --where='primaryCol=1'
31.3.14.5. Restore a table
# create a backup
mysqldump -uroot -p mydb mytable > mytable.sql
mysql -uroot -p mydb < mytable.sql

# From a full database backup

# Method 1: From a full database backup, create a temporary db
# - make sure dbBackup.sql does not have `use mydb`. phpMyAdmin might have it but mysqldump won't have it
mysql -uroot -p -e "create database temporary"
mysql -uroot -p temporary < dbBackup.sql
mysqldump -uroot -p temporary mytable > mytable.sql
mysql -uroot -p mydb < mytable.sql
mysql -uroot -p -e "drop database temporary"

# Method 2: From a full database backup, extract a table from a db backup
sed -n -e '/-- Table structure for table `abc`/,/UNLOCK TABLES/p' dbBackup.sql > mytable.sql
31.3.14.6. Release space
# See what tables without row data are dumped
mysqldump --single-transaction --routines --triggers --events -uroot -p --all-databases > backup.sql

# --all-databases, -A :: `mysql` DB is exported but not `performance_schema`
# --databases, -B :: `-B mydb1 mydb2`

# Don't export `mysql` nor `performance_schema` DB's
mysqldump --single-transaction --routines --triggers --events -uroot -p -B mydb1 mydb2 > backup.sql

sed -i "s/ENCRYPTION='Y'//g" backup.sql

mysqladmin -uroot -p drop mydb1 drop mydb2

mysql -uroot -p -e 'SET GLOBAL innodb_fast_shutdown = 0'

mysqladmin shutdown
rm -f /var/lib/mysql/ib{data1,_logfile*}
service mysql start

mysqladmin -uroot -p create mydb1
mysqladmin -uroot -p create mydb2

mysql -uroot -p < backup.sql

31.3.15. Binary Log

https://dev.mysql.com/doc/internals/en/binary-log-overview.html

See if it's enabled and see if the format :: ROW, MIXED or STATEMENT 

select variable_value as "BINARY LOGGING STATUS (log-bin) :: "
from information_schema.global_variables where variable_name='log_bin';

select variable_value as "BINARY LOG FORMAT (binlog_format) :: " 
from information_schema.global_variables where variable_name='binlog_format'

// 
show variables like 'datadir';

To update the setup, you need to update the MySQL configuration file (my.ini or my.cnf) by adding the following lines to the [mysqld] section (the Server Section). 

log-bin=bin.log
log-bin-index=bin-log.index
max_binlog_size=100M
binlog_format=row
socket=mysql.sock

show variables like 'datadir';

// Log file location :: datadir/bin.0001.log

show binary logs;

https://jinyuwang.weebly.com/for-mysql/how-to-enable-binary-logging-for-mysql

Recover from deleted rows 

mysqlbinlog binary_log_file > query_log.sql

Then search for missing rows.

31.3.16. phpMyAdmin

https://www.phpmyadmin.net/downloads/ Just put sql-admin to any Wordpress website root directory and run http://yourwebsiteip:port/sql-admin You don't need to do anything else.

It first loads sql-admin\libraries\config.default.php and then load sql-admin\config.inc.php To create a config.inc.php, just duplicate sql-admin/config.sample.inc.php 

/* Authentication type */
// Use cookie
//$cfg['Servers'][$i]['auth_type'] = 'cookie';
// If encounter "Failed to set session cookie. Maybe you are using HTTP instead of HTTPS", use http
$cfg['Servers'][$i]['auth_type'] = 'http';
/* Server parameters */
// change host
$cfg['Servers'][$i]['host'] = 'database';
$cfg['Servers'][$i]['compress'] = false;
$cfg['Servers'][$i]['AllowNoPassword'] = false;

31.3.17. Adminer db:adminer

When choose to dump (export in newer version) a database, leave blank for Database (don't DROP+CREATE) but DROP+CREATE for tables

Routines and Events for Database Only Triggers for Tables Data :: INSERT 

-- START We don't want these lines
DROP DATABASE IF EXISTS `your_db`;
CREATE DATABASE `your_db`
USE `your_db`
-- END We don't want these lines

DROP TABLE IF EXISTS `wp_abc`;
CREATE TABLE `wp_ABC` ...

31.3.18. UC: Concatenate values of a column across multiple rows about the same record

  • mysqlfiddle

  • Syntax 

    GROUP_CONCAT([DISTINCT] expr [,expr ...]
     [ORDER BY {unsigned_integer | col_name | expr}
         [ASC | DESC] [,col_name ...]]
     [SEPARATOR str_val])
  • Without SEPARATOR, default is comma ,
    • GROUP_CONCAT(col1)
  • GROUP_CONCAT only concatenates non-null values. If all values are null, null will be returned

  • GROUP_CONCAT by default only handles 1024 letters. Change it for the current session 

    SET SESSION group_concat_max_len = 65535;

SELECT s.id, s.firstname, s.lastname
, GROUP_CONCAT(DISTINCT ts.score ORDER BY ts.testDate SEPARATOR ', ') AS student_scores
, GROUP_CONCAT(DISTINCT liConCat2(c.course, c.year) SEPARATOR 0x1E) AS courses
FROM student s 
     LEFT JOIN s_score ts ON ts.sid = s.id
     LEFT JOIN s_course c ON c.sid = s.id
GROUP BY s.id
ORDER BY s.lastname

If there're multiple fields that have multiple values, e.g. a student can have multiple scores and multiple courses, and they are joined,

Total number of rows are for one student is: N(courses of student) X N(scores of student), N(x) is number of x, when x is zero, N(0) = 1.

When you do GROUP_CONCAT on course, use DISTINCT inside.

Concatenate multiple fields GROUP_CONCAT(DISTINCT liConCat2(c.course, c.year) SEPARATOR 0x1E) AS courses

31.3.18.1. Concatenate strings, use a different separator
  • 0x1F (31): unit (fields) separator
  • 0x1E (30): record separator
  • 0x1D (29): group separator
GROUP_CONCAT(foo SEPARATOR 0x1D)
$array = explode(chr(29),$s);
31.3.18.2. JSON_ARRAYAGG
  • Return a string of JSON or NULL
  • Pros
  • Cons
    • Since MySQL 5.7.22
    • DISTINCT is not allowed and other expressions are allowed
SELECT s.sid, s.firstname, s.lastname
  , JSON_ARRAYAGG(ts.final_exam_score) AS student_scores
FROM student s
       LEFT JOIN s_score ts ON ts.ssid = s.sid
       LEFT JOIN s_course c ON c.sid = s.sid
GROUP BY s.sid
ORDER BY s.lastname

31.3.19. UC: Split Comma-Separated Values mysql:split list

Given table dashboards

ID recipients
1 a@x.ca,b@x.ca
2 b@x.ca
3 c@x.ca

Result

email count
b@x.ca 2
a@x.ca 1
c@x.ca 1

#+NAME Create a temporary table 

-- create X rows which is equal or bigger than the max number of items across every `recipients` column.
-- e.g. row 1 has 2 items, row 2 and 3 has 1 item
-- Max number of items across all rows is then 2
-- A temporary table will have a column named `n` and it has 2 rows
create temporary table numbers as (
  select 1 as n
  union select 2 as n
  union select 3 as n
  -- ...
)

A temporary table numbers is created with 2 rows

n
1
2 
SELECT * 
JOIN dashboards
JOIN numbers
  ON char_length(recipients) 
    - char_length(replace(recipients, ',', '')) 
    >= numbers.n - 1

-- ON (number of commas in dashboards.recipients) < (n - 1)

Result

ID recipients n
1 a@x.ca,b@x.ca 1
1 a@x.ca,b@x.ca 2
2 b@x.ca 1
3 c@x.ca 1 
SELECT 
  id, 
  substring_index(
    substring_index(recipients, ',', n), 
    ',', 
    -1
  ) AS email
FROM dashboards
JOIN numbers
  ON char_length(recipients) 
    - char_length(replace(recipients, ',', '')) 
    >= n - 1

Result

ID email
1 a@x.ca
1 b@x.ca
2 b@x.ca
3 c@x.ca

All together 

SELECT email, count(1) FROM ( 
  SELECT 
    id, 
    substring_index(
  substring_index(recipients, ',', n), 
  ',', 
  -1
    ) AS email
  FROM dashboards
  JOIN numbers
    ON char_length(recipients) 
  - char_length(replace(recipients, ',', '')) 
  >= n - 1
) email_recipients_by_dashboard
GROUP BY 1

31.3.20. TS: Unknown collation: utf8mb4_unicode_520_ci mysql:unknown collation

  • Error encountered when importing .sql in phpMyAdmin #1273 - Unknown collation: 'utf8mb4_unicode_520_ci'
    Replace collation and upload .sql again
    sed -i 's/utf8mb4_unicode_520_ci/utf8mb4_unicode_ci/g' file.sql

31.4. MariaDB

31.4.1. Install

sudo apt install mariadb-server
# service will start automatically, check status
sudo systemctl status mariadb

# MariaDB version
mysql -V

# mysql  Ver 15.1 Distrib 10.1.38-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
# 15.1 is the command line client version. 10.1.38-MariaDB is the mysql server version the client was built with
# In order to get the mysql server version, have to do it in console: mysql:config

# improve security: prompt for root user password, remove anonymous user, restrict root user access to local machine and remove the test database
# Y to all
sudo mysql_secure_installation

31.4.2. Config loading

  • Hyphen vs Underscore
    • In most cases, they have the same effect in cofig files. But config variables returned from queries are always underscore
  • not exist in Ubuntu install
  • loads other files. symlink to /etc/alternatives/my.cnf symlink to /etc/mysql/mariadb.cnf
    /etc/mysql/mariadb.cnf
    global defaults. This file
    In MySQL
    /etc/mysql/mysql.cnf
    /etc/mysql/conf.d/*.cnf

    global options

    • In MySQL
    mysql.cnf
    [mysql] empty
    mysqldump.cnf
    [mysqldump] not empty
    /etc/mysql/mariadb.conf.d/*.cnf

    MariaDB-only options

    50-server.cnf
    [mysqld] bind-address
    50-client.cnf
    50-mysqld_safe.cnf
    50-mysql-clients.cnf
    In MySQL
    /etc/mysql/mysql.conf.d/*.cnf
    mysqld.cnf
    [mysqld_safe] [mysqld]
    mysqld_safe_syslog.conf
    [mysqld_safe]
    (no term)
    Custom config files should have only one dot and extension is .cnf
  • by default not created. User-specific options
# Default options are read from the following files in the given order:
mysqld --help --verbose | less

# Usually
/etc/my.cnf /etc/mysql/my.cnf ~/.my.cnf 

# see which defaults (after changes) mysqld is using
mysqld --print-defaults

This does not work with Ubuntu. Change 50-server.cnf instead. 

[mysqld]

#skip-networking
# default is no skip-networking directive
# if it's there, MariaDB will not work with any connection with TCP/IP

bind-address = 127.0.0.1
# default. 127.0.0.1 is also localhost. No one can connect to server from other hosts or from the same host over TCP/IP on a different interface than the loopback (127.0.0.1).
# Which means only local connection and the host should be referred to 127.0.0.1 or localhost.

# remove bind-address the directive by commenting out so that it binds to any IP `0.0.0.0` or `::`
# it becomes with commeting out!

#skip-networking
#bind-address = 127.0.0.1

# or in the last conf file ~/.my.cnf or last in /etc/my.cnf
[mysqld]
skip-networking=0
skip-bind-address

31.4.3. Restart

systemctl enable mariadb
systemctl status mariadb
systemctl start mariadb
systemctl stop mariadb

31.4.4. Manage users

-- see any users that can connect remotely
SELECT User, Host FROM mysql.user WHERE Host <> 'localhost';

31.4.5. CREATE OR REPLACE [PROCEDURE|FUNCTION|TRIGGER]

31.4.6. Trigger

List triggers in all dbs
SHOW TRIGGERS;
List triggers for a db
SHOW TRIGGERS FROM mydbname;
List triggers for a table
SHOW TRIGGERS FROM mydbname LIKE 'mytblname'; or
  • SHOW TRIGGERS FROM mydbname WHERE 'Table' = 'mytblname';
See a specific trigger
SHOW TRIGGERS FROM mydbname WHERE `Trigger` = 'my_trigger_name'\G
Drop a trigger
DROP TRIGGER mydbname.mytriggername;
(no term)
Can call stored procedures but cannot pass NEW nor OLD as the whole row 
DROP TRIGGER IF EXISTS [schema_name.]trigger_name

DELIMITER //
CREATE OR REPLACE TRIGGER trigger_newguid
  BEFORE UPDATE
  ON sp
  FOR EACH ROW
BEGIN
  IF NEW.modifiedon <> OLD.modifiedon THEN
  -- Don't use `UPDATE trigger_newguid SET` because we are in a row
  -- `BEFORE UPDATE` is required otherwise the record/row is locked
    SET NEW.newguid = UuidToBin(uuid());
  end if;
END;
//

DELIMITER ;

-- Update field modifiedon to trigger trigger_newguid
-- newguid field will be updated before the modifiedon field is updated
UPDATE sp SET modifiedon = NOW();

31.4.7. Stored function mariadb:stored function

  • Refer to mariadb:uuid
  • It can't be run in CREATE TABLE ALTER TABLE

31.4.8. UUID mariadb:uuid

DELIMITER //

CREATE FUNCTION UuidToBin(_uuid BINARY(36))
  RETURNS BINARY(16)
  LANGUAGE SQL  DETERMINISTIC  CONTAINS SQL  SQL SECURITY INVOKER
RETURN
  UNHEX(CONCAT(
  SUBSTR(_uuid, 15, 4),
  SUBSTR(_uuid, 10, 4),
  SUBSTR(_uuid,  1, 8),
  SUBSTR(_uuid, 20, 4),
  SUBSTR(_uuid, 25) ));

CREATE FUNCTION UuidFromBin(_bin BINARY(16))
  RETURNS BINARY(36)
  LANGUAGE SQL  DETERMINISTIC  CONTAINS SQL  SQL SECURITY INVOKER
RETURN
  LCASE(CONCAT_WS('-',
          HEX(SUBSTR(_bin,  5, 4)),
          HEX(SUBSTR(_bin,  3, 2)),
          HEX(SUBSTR(_bin,  1, 2)),
          HEX(SUBSTR(_bin,  9, 2)),
          HEX(SUBSTR(_bin, 11))
    ));

//
DELIMITER ;

Then you can 

CREATE TABLE t
(
  id      binary(16) PRIMARY KEY,
  id_text varchar(36) GENERATED ALWAYS AS (
        LCASE(CONCAT_WS('-',
                HEX(SUBSTR(id, 5, 4)),
                HEX(SUBSTR(id, 3, 2)),
                HEX(SUBSTR(id, 1, 2)),
                HEX(SUBSTR(id, 9, 2)),
                HEX(SUBSTR(id, 11))
      ))
        ) virtual
);

-- Letting MySQL create the UUID:
INSERT INTO t (uuid, ...) VALUES (UuidToBin(UUID()), ...);

-- Creating the UUID elsewhere:
INSERT INTO t (uuid, ...) VALUES (UuidToBin(?), ...);

-- Retrieving (point query using uuid):
SELECT ... FROM t WHERE uuid = UuidToBin(?);

-- Use the stored function on the righ hand side in WHERE clause. Don't do this
-- WHERE UuidFromBin(uuid) = '1026-baba-6ccd780c-9564-0040f4311e29'

-- Retrieving (other):
SELECT UuidFromBin(uuid), ... FROM t ...;

31.5. MSSQL

31.5.1. Modify column values if exist (CASE WHEN)

SELECT colOne,
varColTwo = 
CASE colTwo
 WHEN 1 THEN 1
 WHEN 0 THEN 3
END,
varImage = 
CASE
 WHEN len(colImage) > 0 THEN 'http://abc.com/images/' + colImage
 ELSE ''
END
FROM aTable
ORDER BY varColTwo DESC,
CASE 
 WHEN colFour = 1 THEN colFive
 WHEN colFour = 2 THEN colSix
END ASC

31.5.2. Concatenate Multiple Columns to Single Column as String

A doc has multiple categories. Concatenate the categories delimited by commas. 

SELECT doc.intDocID
  ,doc.varTitle
  ,STUFF(
      (SELECT ',' + CAST( t2.catID AS varchar(10) )
       FROM tblDocCat t2
       WHERE t2.intDocID = doc.intDocID
       FOR XML PATH('')
      ),
      1,
      1,
      ''
   ) AS docCats
31.5.2.1. STUFF

This deletes the first letter of a column and returns the rest. STUFF first deletes a substring from aCol at start:1 and length_to_delete:1 and then insert a string at the start position 

STUFF(aCol, 1,1, '')

31.5.3. Export as CSV - BCP mssql:bcp

  • bcp.exe is in C:\Program Files\Microsoft SQL Server\90\Tools\Binn\

  • Options 

    bcp "Select * from fullDatabasename.dbo.tableName" queryout "c:\aFolder\aFile.csv" -c -t"\",\"" -r"\"\n\"" -T
    c
    Ascii
    t
    field terminator. Comma
    r
    row terminator. New line "\n"
    T
    trusted connection
  • DEDP225.[dev.todaystrucking].dbo.[tblDocument]
31.5.3.1. Escape % if bcp is in batch file
bcp "SELECT * FROM aTable WHERE email LIKE '%%@%%.%%'"
31.5.3.2. Save long query in Global Temporary table (GTT)
SELECT *
INTO ##myglobaltemptable
FROM tableA

Drop the GTT after bcp: DROP TABLE ##myglobaltemptable 

bcp ##myglobaltemptable out "c:\folder\file.csv" -c -t"\",\"" -r"\"\n\"" -T
31.5.3.3. Get all columns
DECLARE @colnames VARCHAR(max);
SELECT @colnames = COALESCE(@colnames + ',', '') + char(39)+ column_name +char(39) 
FROM INFORMATION_SCHEMA.COLUMNS where TABLE_NAME='%yourtablename%'; 
SELECT @colnames

It returns 'col1','col2',...,'colLast'

Export to header.csv bcp "SELECT 'col1','col2',...,'colLast'" queryout "c:\header.csv" -c -t"\",\"" -r"\"\n\"" -T

31.5.3.4. Last step

Both header and data files need to be fixed. Insert a double quote at the beginning of the file and delete the last double quote at the end of it. Insert a new line at the end of the header file.

Combine 2 files
copy /b c:\header.csv+c:\folder\file.csv c:\export.csv

31.5.4. Export as XML

  • By default, it's UTF-8 encoded. datetime will be automatically converted to XML format
  • If value is NULL, the xml node will not be generated
  • By default, MSSQL will add <CR><LF> into the Unicode XML data stream every 2033 characters
  • Use mssql:bcp with option -r to add no new line
31.5.4.1. Query
SELECT CustomerID as "@id",
CompanyName,
Address as "address/street",
City as "address/city",
Region as "address/region",
PostalCode as "address/zip",
Country as "address/country",
ContactName as "contact/name",
ContactTitle as "contact/title",
NULLIF(Phone,'') as "contact/phone",
-- If value is NULL, the xml node will not be generated
Fax as "contact/fax"
FROM Customers
FOR XML PATH('Customer'), ROOT('doc')

Result 

<doc>
  <Customer id="ALFKI">
    <CompanyName>Alfreds Futterkiste</CompanyName>
    <address>
  <street>Obere Str. 57</street>
  <city>Berlin</city>
  <zip>12209</zip>
  <country>Germany</country>
    </address>
    <contact>
  <name>Maria Anders</name>
  <title>Sales Representative</title>
  <phone>030-0074321</phone>
  <fax>030-0076545</fax>
    </contact>
  </Customer>
  ...
</doc>

INNER JOIN Multiple records 

SELECT d.docID
,(
SELECT c.category AS category
FROM tblDocCats dc
INNER JOIN tblCategory c ON c.catID = dc.catID
WHERE dc.docID = d.docID
FOR XML PATH(''), TYPE
) AS categories
FROM doc d
FOR XML PATH('item'), ROOT('items')

Result 

<items>
 <item>
  <docID></docID>
  <categories>
   <category></category>
   <!--...-->
  </categories>
 </item>
</items>

This 

SELECT d.docID
,(
SELECT c.category AS name, c.catID as ID
FROM tblDocCats dc
 INNER JOIN tblCategory c ON c.catID = dc.catID
WHERE dc.docID = d.docID
FOR XML PATH('category'), TYPE
) AS category
FROM doc d
FOR XML PATH('item'), ROOT('items')

will result as 

<items>
  <item>
    <docID></docID>
    <category>
  <name></name>
  <id></id>
    </category>
    <category>
  <name></name>
  <id></id>
    </category>
    ...
  </item>
</items>
31.5.4.2. Different modes

FOR XML AUTO 

<servername.dbo.Customers ID="C11" Name="..." Address="..." />

FOR XML RAW 

<row ID="C11" Name="..." Address="..." />

FOR XML PATH 

<row>
  <ID></ID>
  <Name></Name>
  <Address></Address>
</row>
31.5.4.3. bcp
bcp "query..." queryout C:\file.xml -T -w -r
r no value means don't add any newline character for every 2033 characters or new row
w Use Unicode characters
S server_name[\instance_name]

Save long query as stored procedure. You can't save FOR XML in Global Temporary Table. dev.todaystrucking is the database name. Inside the real SELECT, you can refer to table as dbo.tablename not DEDP225.[dev.todaystrucking].dbo.tablename 

USE [dev.todaystrucking]
GO

IF OBJECT_ID('dbo.usp_exporttr') IS NOT NULL DROP PROC dbo.usp_exporttr
GO

CREATE PROC dbo.usp_exporttr AS

SET NOCOUNT ON

SELECT ...
FOR XML PATH('item'), ROOT('items')

RETURN
GO

Then creat another query in MS SQL Server Management Studio and run 

EXEC xp_cmdshell 'bcp "EXEC [dev.todaystrucking].dbo.usp_exporttr" queryout "c:\a.xml" -S (local) -T -w -r'

You may receive "SQL Server blocked access to procedure 'sys.xp_cmdshell' of component 'xp_cmdshell' because this component is turned off as part of the security configuration"

Just go to bcp folder and run as a command line mssql:bcp 

bcp "EXEC [dev.todaystrucking].dbo.usp_exporttr" queryout "c:\a.xml" -S (local) -T -w -r

The exported xml file will have UCS-2 encoding which is a preceding version of UTF-16. This encoding is in BOM. For SQL Server lower than 2016, you can't set encoding to UTF-8.

31.5.4.4. bcp clean up

You can run these in Git Bash on Windows Change encoding to UTF-8 iconv -f UCS-2 -t UTF-8 a.xml > b.xml

Add xml declaration echo '<?xml version="1.0"?>' | cat - b.xml > c.xml

Delete xml:invalid_characters sed sed "s/&#x1f;//gi; s/&#x1e;//gi;" c.xml > d.xml

31.5.5. Copy to Excel

C-a and C-c, C-v in Excel For datetime, you can =now() in a new cell, format paint the datetime columns.

In the =now() cell, you can check out the Custom format. It's yyyy-mm-dd h:mm You can change it to yyyy-mm-dd h:mm:ss AM/PM

31.5.6. Migrate to MySQL

Export MSSQL to a bak file. mssql:backup https://stackoverflow.com/questions/156279/how-to-import-a-sql-server-bak-file-into-mysql

If the bak file is less than 10gb, follow this: https://www.silicongadget.com/software/database/import-mssql-bak-files-to-mysql/2955/

  • Restore MSSQL database using the bak file using MS SQL Server Express Edition
  • Use MySQL Workbench to connect to the MSSQL database and then migrate to MySQL

31.5.7. Connections

Number of active connections by database 

SELECT 
    DB_NAME(dbid) as DBName, 
    COUNT(dbid) as NumberOfConnections,
    loginame as LoginName
FROM sys.sysprocesses
WHERE dbid > 0
GROUP BY dbid, loginame

-- 2017 new query

SELECT @@ServerName AS server
 ,NAME AS dbname
 ,COUNT(STATUS) AS number_of_connections
FROM sys.databases sd
LEFT JOIN sys.sysprocesses sp ON sd.database_id = sp.dbid
GROUP BY NAME

31.5.8. Last access

SELECT d.name,
last_user_seek = MAX(last_user_seek),
last_user_scan = MAX(last_user_scan),
last_user_lookup = MAX(last_user_lookup),
last_user_update = MAX(last_user_update)
FROM sys.dm_db_index_usage_stats AS i
JOIN sys.databases AS d ON i.database_id=d.database_id
GROUP BY d.name

31.5.9. Backup

mssql:backup https://docs.microsoft.com/en-us/sql/relational-databases/backup-restore/create-a-full-database-backup-sql-server Export database to bak file. SQL Management Studio Tool > right click on a database and Tasks > Backup > Use Full backup, specify a destination (path to bak). Use overwrite settings in Options. Then click on Script > Script to file. You will get a .sql: 

BACKUP DATABASE [aDatabase] TO DISK ='N'C:\afile.bak' WITH NOFORMAT, INIT, 
NAME = N'aDatabase-Full Database Backup', 
SKIP, NOREWIND, NOUNLOAD,  STATS = 10
GO

Make filename dynamic (remove spaces if necessary): 

DECLARE @filename nvarchar(200) = ''
SELECT @filename = 'C:\test-' + convert(varchar(23), getdate(), 126) + '.bak'
BACKUP DATABASE [aDatabase] TO
 DISK =  @filename WITH NOFORMAT, INIT,  
 NAME = N'aDatabase-Full Database Backup', SKIP, NOREWIND, NOUNLOAD,  STATS = 10
GO

Run this .sql file: 

sqlcmd -S sqlservername -i C:\sqlFileName.sql

31.6. MongoDB

31.7. GraphQL

GitHub GraphQL API explorer
https://developer.github.com/v4/explorer/

31.8. ClickHouse

  • Refer to docker:clickhouse
  • Use uppercase for functions
  • INSERT INTO () VALUES (), don't use VALUE

32. Google Ad Manager - DFP

32.1. Default code - Google Publisher Tag - gpt.js

<head>
    <script async='async' src='https://securepubads.g.doubleclick.net/tag/js/gpt.js'></script>
    <script>
  var googletag = googletag || {};
  googletag.cmd = googletag.cmd || [];
    </script>

    <script>
  googletag.cmd.push(function() {
    googletag.defineSlot('/123456/my_ad_unit_1', [300, 250], 'div-gpt-ad-1557260744066-0').addService(googletag.pubads());
    googletag.pubads().enableSingleRequest();
    googletag.enableServices();
  });
    </script>
</head>

<body>
<!-- /123456/my_ad_unit_1 -->
<div id='div-gpt-ad-1557260744066-0' style='height:250px; width:300px;'>
    <script>
  googletag.cmd.push(function() { googletag.display('div-gpt-ad-1557260744066-0'); });
    </script>
</div>
</body>

32.1.1. Legacy

<head>
    <script>
    var googletag = googletag || {};
    googletag.cmd = googletag.cmd || [];
    (function () {
        var gads = document.createElement('script');
        gads.async = true;
        gads.type = 'text/javascript';
        var useSSL = 'https:' == document.location.protocol;
        gads.src = (useSSL ? 'https:' : 'http:') +
        '//www.googletagservices.com/tag/js/gpt.js';
        var node = document.getElementsByTagName('script')[0];
        node.parentNode.insertBefore(gads, node);
    })();

    googletag.cmd.push(function () {

        googletag.defineSlot('/123456/my_ad_unit_1',
        [300, 250], 'div-gpt-ad-unit-1')
        .addService(googletag.pubads());

        googletag.pubads().collapseEmptyDivs();
        googletag.pubads().enableSingleRequest();
        googletag.enableServices();
    });
    </script>
</head>
<body>
<!-- /123456/my_ad_unit_1 -->
<div id='div-gpt-ad-unit-1' style='height:250px; width:300px;'>
    <script>
    googletag.cmd.push(function() { googletag.display('div-gpt-ad-unit-1'); });
    </script>
</div>
</body>

32.2. Why async is required

  • Video companion requires async
  • pubService.refresh requires async
  • POST request requires async. GET request with larger than 2048 bytes not supported
  • However, it's better to use sync mode for rich media creative unless the creative is in friendly frames

32.3. Competition Calculation

  • List of candidate line items
  • select the best line item
  • select the best creative associated with the winning line item
  • DFP remnant and Ad Exchange/Adsense line items are evaluated

to see if they have line items with a higher yield

  • the best creastive associated with the winning line item is selected
  • creative returned to the browser. Dynamic allocation.

32.4. Targeting - key value, Geo

32.4.1. Key-value

Filter format
AND(site=MySiteA,NOT(exclusive=exclusive))

Slot-level targeting is recommended 

googletag.defineSlot('/123/travel/asia/food', [728, 90], "div-id")
     .addService(googletag.pubads())
     .setTargeting("interests", ["sports", "music"]);

Page-level targeting 

googltag.pubads().setTargeting("topic", "basketball");

32.4.2. Geolocation

Postcal code
target Canada to the first 3 characters which is Forward Sortation Area (the last 3 are called Local Delivery Unit). The first character can show which province. Big provinces like ON and QC have multiple first characters
City, province, country
Some big city has multiple cities grouped together e.g. Vaughan city area has Concord, Maple, Thornhill and Woodbridge cities

32.5. Exclusive Ads on Specific Page

Let's say there're 2 LBs and 2 BBs throughout the website. 999/LB1, 999/LB2, 999/BB1, 999/BB2 For a specific page, Line Item A takes over all ad units. For another specific page, Line Item B takes over all ad units.

The perfect way Create multilevel ad units and make them Special Ad Units. 999/LB1/exclusive, 999/LB2/exclusive 999/BB1/exclusive, 999/BB2/exclusive

On those specific pages:

  • Use the new tags of multilevel and special ad units on HTML instead of the original ones.
  • setTargetting on HTML and on DFP so that Line Item A and B can be distinguished.

The better way Multilevel and Special ad units are only avaiable in Premium. Here's the way for SBS

  • 999/LB1_exclusive, 999/LB2_exclusive
  • 999/BB1_exclusive, 999/BB2_exclusive
  • Use the new tags on HTML instead of the original ones
  • setTargetting on HTML and DFP so that Line Item A and B

The workaround way If you don't have control on HTML, you can:

  • For Line Item A and B
    • On DFP
      • add Key-Value pair exclusive is exclusive
      • add Key-Value pair sponsor is lineitemA
    • In HTML, for those specific pages
      • setTargeting("exclusive","exclusive");
      • setTargeting("sponsor","lineitemA");
  • For other normal line items
    • On DFP
      • add Key-Value par 'exclusive is not exclusive' ( means contain and * means begin with)
    • In HTML, do nothing (remain the same)

32.6. Email - Non-JavaScript

Tagless Request
https://admanager.google.com/95740733#delivery/line_item/detail/order_id=2556961137&line_item_id=5091153061

  • Was called Simply URL 

      <!-- Email HTML -->
  <a href="http://my.com/ad/ad_unit_name/728x90/">
<img width="728" height="90" alt=""
     src="https://securepubads.g.doubleclick.net/gampad/ad?iu=/networkid/ad_unit_name&sz=728x90&c=[todays_date][account]" />
  </a>

  <!-- Website -->
  <script src="//code.jquery.com/jquery-1.10.2.js"></script>
  <script>
var ad_unit = 'ad_unit_name';
var ad_unit_size = '728x90';
var rnd=Math.floor(Math.random() * (999999 - 10 +1)) + 10;
var theUrl='//pubads.g.doubleclick.net/gampad/adx?iu=/' + ad_unit 
+ '&sz=' + ad_unit_size + '&c=' + rnd;
var xmlHttp = null;
xmlHttp = new XMLHttpRequest();
xmlHttp.open( "GET", theUrl );
xmlHttp.send();
xmlHttp.onreadystatechange=function() {
if (xmlHttp.readyState==4 && xmlHttp.status==200) {
xmlDoc = $.parseHTML( xmlHttp.response );
adlink = $(xmlDoc).find('a').attr('href');
if (typeof adlink == "undefined") adlink = "http://www.example.com";
window.location = adlink;
}
}
  </script>

    PHP, DFP creative type Custom. Get response line by line 

      $url = "http://pubads.g.doubleclick.net/gampad/adx?iu=/12343/ad_unit_name&sz=230x20&c=". REQUEST_TIME."&m=text/html";
  $json = @file_get_contents($url);
  if ($json !== FALSE) {
$lines = explode("\n", $json);
$link = ( isset($lines[0]) ) ? trim($lines[0]) : 'default';
  }

    WordPress example: wp:filter:template_include GitHub Gist: https://gist.github.com/levonlee/888f5f5b0a33acebe2b20ecbbff3b106

Base URL
https://securepubads.g.doubleclick.net/gampad/[request-type]?[parameters]
Before
https://pubads.g.doubleclick.net/gampad/[request-type]?[parameters] or http://
(no term)
Request Type
ad
simple image creative
adx
return raw code of creative which is to be placed inside an iframe with JS available
jump
log click and redirect to destination URL
clk
only log click but no redirect. Only in DFP Premium
(no term)

Parameters

iu for all request types
sz for ad, jump and adx
t for ad, jump and adx. key-value pairs
excl_cat competitive exclusions
c for ad, jump and adx. cache buster. only number
m for adx. Specify file type to return. m=text/html
submodel for all request types. Mobile device name
u_w for all request types. Mobile device screen height.
title required if mutliple ad tags use the same ad unit code on the same page
mob Indicaste it's a mobile request.
(no term)
Delay impression counting. Choose one

  • Add URL parameters &d_imp=1&d_imp_hdr=1 in the requests (ad or adx)

    d_imp
    if enabled, impression counting upon request is disabled
    d_imp_hrd
    if enabled, the http response header contains the url needed to ping
    Google-Delayed-Impression response header
    Ad Manager impression URL
    Google-3rdParty-Delayed-Impression response header
    3rd party impression URL, if present in creative
    Ping the URL with http request header Google-Delayed-Impression
    an "Ad server impression" and "Ad server downloaded impression" is recorded
    Note
    ad request does not return anything. Use adx request

32.7. Responsive

  • Add a size to the ad unit /networkid/adunitname
  • Specify one or more sizes for a line item that uses this ad unit
  • Upload each creative and specify the size. You can have creative bigger than the size you specify
googletag.cmd.push(function() {
  var leaderboard_mapping = googletag.sizeMapping()
  .addSize([1280, 800], [[970, 250], [970, 90], [728, 90]])
  .addSize([600, 800], [468, 60]) // viewport size is width and height both > 600 and 800
  .addSize([0, 0], [[320, 50], [320, 100]]) // Map any viewport size
/* This will only display the ad if it's mobile or tablet viewport size
  .addSize([0, 0], []) // [] means don't try to call an ad
  .addSize([320, 700], [300, 250]) // Mobile or tablet
  .addSize([1050, 200], [])
*/
  .build();

  googletag.defineSlot('/networkid/adunitname'
       , [728, 90] // If there is an error in the mapping or if the browser size can't be determined
           // the size specified here in .defineSlot will be used. 
       , 'div-id')
       .defineSizeMapping(leaderboard_mapping)
       .addService(googletag.pubads());
});

32.8. Macro in Third Party Code, Custom Code

CACHEBUSTER
time in integer
CLICK_URL_UNESC
normal click macro e.g.
  • %%CLICK_URL_UNESC%%https://adserver.ca/destination
  • %%CLICK_URL_UNESC%%%%DEST_URL%%
CLICK_URL_ESC
escaped click macro (&, ? , %). The macro is passed as a parameter to 3rd party server
  • https://adserver.ca/destination?ncu=%%CLICK_URL_ESC%%
VIEW_URL_UNESC, VIEW_URL_ESC
required in Custom Code
  • %%VIEW_URL_UNESC%%%%FILE:JPG1%%
(no term)

Example dfp:third party code Third Party Code with macro automatically inserted 

<!-- url parameter ncu is inserted, ord's [timestamp] is replaced by CACHEBUSTER -->
<script src="http://bs.serving-sys.com/Serving/adServer.bs?c=28&cn=display&pli=1074030166&w=728&h=90&ncu=$$%%CLICK_URL_UNESC%%$$&ord=%%CACHEBUSTER%%&ifrm=-1&z=0"></script>
<noscript>
<!-- insert CLICK_URL_UNESC in href -->
<a href="%%CLICK_URL_UNESC%%http://bs.serving-sys.com/Serving/adServer.bs?cn=brd&pli=1074030166&Page=&Pos=944448101" target="_blank">
<img src="http://bs.serving-sys.com/Serving/adServer.bs?c=8&cn=display&pli=1074030166&Page=&Pos=944448101" border=0 width=728 height=90></a>
</noscript>

dfp:custom code

  • VIEW_URL_UNESC or VIEW_URL_ESC should be included otherwise impression won't be tracked.
  • In preview, this view macro returns empty but that's normal. It has value when the creative is served.
  • Sometimes you need to have DFP inserted the macros in third party code mode, and then use that in Custom Code.

32.9. Creative types

32.9.1. Creative Type: Third Party Code - Video

  • Video is hosted on third party server
  • Video player size often doesn't match the ad unit size e.g. big box

  • Creative examples on DFP. Refer to video:jw 

    <!-- JW script tag -->
<script src="//content.jwplatform.com/players/MEDIAID-PLAYERID.js?v=%%CACHEBUSTER%%"></script>

<!-- JW iframe tag -->
<div style="position:relative; padding-bottom:56.25%; overflow:hidden;"><iframe src="https://cdn.jwplayer.com/players/MEDIAID-PLAYERID.html?v=%%CACHEBUSTER%%" width="100%" height="100%" frameborder="0" scrolling="auto" allowfullscreen style="position:absolute;"></iframe></div>

<!-- Viewbix -->
<iframe width="300" height="250" 
src="http://www.viewbix.com/frame/1234-567-890?w=300&h=250&ord=%%CACHEBUSTER%%" 
frameborder="0" scrolling="no" allowTransparency="true"></iframe>

32.9.2. Creative Type: HTML5

Upload zip file or single html file

32.9.3. Creative Type: DoubleClick Tag, DCM Tag

DCM, previously known as DFA (for advertisers), is DoubleClick Campaign Manager, part of the DoubleClick Marketing Solutions.

DCM placement tags

Always use Internal redirect tag for DFP. Sometimes this tag is not given by agency, you will need to convert it to this type. Ins tag is recommended for ad servers other than DFP because it has active view capability.

ddm/jump tag (redirect to final destination) has to be loaded with ddm/ad tag.

Other tags like ddm/trackimp, ddm/trackclk, ddm/clk don't have to be loaded with ddm/ad tag.

32.9.3.1. Standard tags (ddm/jump, ddm/ad)
<A HREF="https://ad.doubleclick.net/ddm/jump/Nxxxx.site-keyname/Byyyyyyy.Pzzzz;sz=widthxheight;kw=[keyword];ord=[timestamp]?">

<IMG SRC="https://ad.doubleclick.net/ddm/ad/Nxxxx.site-keyname/Byyyyyyy.Pzzzz;sz=widthxheight;ord=[timestamp];dc_lat=N;dc_rdid=Czzzz;tag_for_child_directed_treatment=I?" BORDER=0 WIDTH=X HEIGHT=Y ALT="Click Here"></A>

Nxxx: DCM account id Byyyy: DCM campaign ID .Pzzzz: DCM placement ID dc_lat=N: key-value pair for mobile app to pass if the user has enabled "Limit Ad Tracking" dc_rdid=Czzz: key-value pair for mobile app to pass resettable device identifiers in the form of IDFA for iOS or advertising ID (AdID) for Android tag_for_child_directed_treatment=I: key-value pair for mobile app to pass info about if a request may come from a user under the age of 13

32.9.3.2. iframe/javascript tags (ddm/adi, ddm/adj)
<IFRAME SRC=""https://ad.doubleclick.net/ddm/adi/N1234.123456yoursite.com/B1234567.123456789;sz=300x250;ord=[timestamp];dc_lat=;dc_rdid=;tag_for_child_directed_treatment=?"" WIDTH=300 HEIGHT=250 MARGINWIDTH=0 MARGINHEIGHT=0 HSPACE=0 VSPACE=0 FRAMEBORDER=0 SCROLLING=no BORDERCOLOR='#000000'>
  <SCRIPT language='JavaScript1.1' SRC=""https://ad.doubleclick.net/ddm/adj/N1234.123456yoursite.com/B1234567.123456789;abr=!ie;sz=300x250;ord=[timestamp];dc_lat=;dc_rdid=;tag_for_child_directed_treatment=?"">
  </SCRIPT>
  <NOSCRIPT>
    <A HREF=""https://ad.doubleclick.net/ddm/jump/N1234.123456yoursite.com/B1234567.123456789;abr=!ie4;abr=!ie5;sz=300x250;ord=[timestamp]?"">
  <IMG SRC=""https://ad.doubleclick.net/ddm/ad/N1234.123456yoursite.com/B1234567.123456789/B10762433.143897886;abr=!ie4;abr=!ie5;sz=300x250;ord=[timestamp];dc_lat=;dc_rdid=;tag_for_child_directed_treatment=?"" BORDER=0 WIDTH=300 HEIGHT=250 ALT=""Advertisement""></A>
  </NOSCRIPT>
</IFRAME>
32.9.3.3. javascript (ddm/adj)
<SCRIPT language='JavaScript1.1' SRC=""https://ad.doubleclick.net/ddm/adj/N1234.123456yoursite.com/B1234567.123456789;sz=300x250;ord=[timestamp];dc_lat=;dc_rdid=;tag_for_child_directed_treatment=?"">
</SCRIPT>
<NOSCRIPT>
<A HREF=""https://ad.doubleclick.net/ddm/jump/N1234.123456yoursite.com/B1234567.123456789;sz=300x250;ord=[timestamp]?"">
<IMG SRC=""https://ad.doubleclick.net/ddm/ad/N1234.123456yoursite.com/B1234567.123456789;sz=300x250;ord=[timestamp];dc_lat=;dc_rdid=;tag_for_child_directed_treatment=?"" BORDER=0 WIDTH=300 HEIGHT=250 ALT=""Advertisement""></A>
</NOSCRIPT>
32.9.3.4. Pre-fetch tags (ddm/pfadx)
# VAST 2.0
https://ad.doubleclick.net/ddm/pfadx/Nxxxx.site-keyname/Byyyyyyy;kw=[keyword]; sz=widthxheight;ord=[timestamp];dc_lat=N;dc_rdid=Czzzz;tag_for_child_directed_treatment=I;dcmt=text/xml

# VAST 3.0
https://ad.doubleclick.net/ddm/pfadx/Nxxxx.site-keyname/Byyyyyyy;kw=[keyword]; sz=widthxheight;ord=[timestamp];dc_lat=N;dc_rdid=Czzzz;tag_for_child_directed_treatment=I;dcmt=text/xml;dc_vast=3

# VAST 4.0
https://ad.doubleclick.net/ddm/pfadx/Nxxxx.site-keyname/Byyyyyyy;kw=[keyword]; sz=widthxheight;ord=[timestamp];dc_lat=N;dc_rdid=Czzzz;tag_for_child_directed_treatment=I;dcmt=text/xml;dc_vast=4
32.9.3.5. Click tracker (ddm/clk)
https://ad.doubleclick.net/ddm/clk/[ad ID];[placement ID];[verifier]?[click-through URL]
32.9.3.6. Internal redirect tag

https://ad.doubleclick.net/ddm/ad/N1234.123456yoursite.com/B1234567.123456789;sz=320x548 

# Image URL
https://ad.doubleclick.net/ddm/ad/Nxxxx.site-keyname/Byyyyyyy.n;sz=widthxheight;dc_expa=URL

# Click-through URL
https://ad.doubleclick.net/ddm/jump/Nxxxx.site-keyname/Byyyyyyy.n;sz=widthxheight

dc_expa: Ping an encoded URL in order to track real-time expansions of rich media display expanding creatives.

Grab the attribute data-dcm-placement value from either iframe or Ins tag and also the size 

https://ad.doubleclick.net/ddm/ad/[data-dcm-placement];sz=widthxheight
32.9.3.7. Ins tag

In DFP, always use internal redirect tag. May also deploy DCM tag as a custom creative by adding DFP macros. Instructions 

Iframe
<ins class='dcmads' style='display:inline-block;width:300px;height:250px'
    data-dcm-placement='N1234.123456yoursite.com/B1234567.123456789'
    data-dcm-rendering-mode='script'
    data-dcm-https-only
    data-dcm-resettable-device-id=''
    data-dcm-app-id=''>
  <script src='https://www.googletagservices.com/dcm/dcmads.js'></script>
</ins>

Javascript
<ins class='dcmads' style='display:inline-block;width:300px;height:250px'
    data-dcm-placement='N1234.123456yoursite.com/B1234567.123456789'
    data-dcm-rendering-mode='script'
    data-dcm-https-only
    data-dcm-resettable-device-id=''
    data-dcm-app-id=''>
  <script src='https://www.googletagservices.com/dcm/dcmads.js'></script>
</ins>

32.10. Examples

32.10.1. Dynamic load new containers

googletag.cmd.push(function() {
    var n = 'div-id-my-slot-name'; // slot name
    var s = googletag.defineSlot('/95740733/float_comp_1', [300, 250], n).addService(googletag.pubads()); // slot
    googletag.display(n);

    // if previously in header, before enableServices, disableInitialLoad() is used

    // googletag.pubads().disableInitialLoad();
    // googletag.enableServices();

    // then follow first .display and refresh
    // googletag.pubads().refresh([s]);

};

32.10.2. Wallpaper, Site Skin

On DFP 

<script type="text/javascript">
  (function() {
  'use strict';
  parent.jQuery(document).ready(function() {
  var backgroundAdwidth = 480; // TODO: find out the background image width
  var backgroundAdheight = 860; // TODO: find out the background image height
  var backgroundAdLeftImage = '%%VIEW_URL_UNESC%%%%FILE:JPG1%%';
  var backgroundAdRightImage = '%%VIEW_URL_UNESC%%%%FILE:JPG2%%';
  var backgroundAdLinkLeft = '%%CLICK_URL_UNESC%%%%DEST_URL%%';
  var backgroundAdLinkRight = backgroundAdLinkLeft;
  var backgroundAdMainContentDivWidth = 996;
  var backgroundAdExtraTopSpace = 100; // Set to 0 if you don't want extra top space
  parent.jQuery('body').prepend('<div id="background-ad-right"></div><div id="background-ad-left"></div>');
  var jsLink1 = "<script type='text/javascript'>" +
    "var backgroundAdwidth = " + backgroundAdwidth + ";" +
    "var backgroundAdheight = " + backgroundAdheight +";" +
    "var backgroundAdMainContentDivWidth = " + backgroundAdMainContentDivWidth + ";" +
    "var backgroundAdExtraTopSpace = " + backgroundAdExtraTopSpace + ";" +
    "</scr" + "ipt>";
parent.jQuery('head').append(jsLink1);
var jsLink2 = parent.jQuery("<script type='text/javascript' src='http://yours.com/js/backgroundskinad.js'>");
  parent.jQuery('head').append(jsLink2);
  var cssLink = parent.jQuery("<link rel='stylesheet' href='http://yours.com/css/media-w1200-w992.css'>");
  parent.jQuery('head').append(cssLink);
  parent.jQuery('#background-ad-left, #background-ad-right').css({
  'position': 'fixed',
  'z-index': '1',
  'cursor': 'pointer',
  'height': backgroundAdheight
  });
  parent.jQuery('#background-ad-left').css({
  'background': 'url(' + backgroundAdLeftImage + ') no-repeat right top'
  });
  parent.jQuery('#background-ad-right').css({
  'background': 'url(' + backgroundAdRightImage + ') no-repeat left top'
  });
  parent.jQuery('#background-ad-left').click(function() {
  window.open(backgroundAdLinkLeft);
  });
  parent.jQuery('#background-ad-right').click(function() {
  window.open(backgroundAdLinkRight);
  });
  });
  })();
</script>

On Web Server - backgroundskinad.js 

jQuery(document).ready(function($) {
  var backgroundAdMainContentDivWidth = 996;
  var backgroundAdExtraTopSpace = 0;
  function setBackgroundAdPositionX() {
    var adPosition = new Array();
    var viewportWidth = $(window).width();
    var leftSpace = (viewportWidth-backgroundAdMainContentDivWidth)/2;
    if (leftSpace > 0) {
  if (leftSpace <= backgroundAdwidth) {
    adPosition['adWidthRealtime'] = leftSpace;
    adPosition['adHorizontalSpacing'] = 0;
  } else {
    adPosition['adWidthRealtime'] = backgroundAdwidth;
    adPosition['adHorizontalSpacing'] = leftSpace - backgroundAdwidth;
  }
    }
    else {
  adPosition['adWidthRealtime'] = 0;
  adPosition['adHorizontalSpacing'] = 0;
    }

    $('#background-ad-left').css({
  'left': adPosition['adHorizontalSpacing'],
  'width': adPosition['adWidthRealtime']
    });

    $('#background-ad-right').css({
  'right': adPosition['adHorizontalSpacing'],
  'width': adPosition['adWidthRealtime']
    });
    return adPosition;
  }

  function setBackgroundAdPositionY() {
    var topspace = backgroundAdExtraTopSpace - $(window).scrollTop();
    // nav HTML element height
    if ($('#newcom-header-masthead').length && $('#newcom-header-masthead').height()) {
  topspace += $('#newcom-header-masthead').height();
    }
    topspace = (topspace > 0) ? topspace : 0;
    $('#background-ad-left, #background-ad-right').css('top', topspace + 'px');
  }

  function wallpaperFixViewport() {
    $('.cover-slogan, .cover-logo').addClass("m-clear");
    $('#newcom-single-left').toggleClass("col-md-9 col-md-8");
    $('#newcom-single-right').toggleClass("col-md-3 col-md-4");
    $('#newcom-page-news-middle').toggleClass("col-md-7 col-md-6");
    $('#newcom-page-news-right').toggleClass("col-md-3 col-md-4");
    $('#newcom-taxonomy-videoseries-middle').toggleClass("col-md-7 col-md-6");
    $('#newcom-content-sidebar-page-left').toggleClass("col-md-9 col-md-8");
    $('#newcom-content-sidebar-page-right').toggleClass("col-md-3 col-md-4");
    $('#newcom-page-middle').toggleClass("col-md-7 col-md-6");
    $('#newcom-page-right').toggleClass("col-md-3 col-md-4");
    $('#newcom-search-middle').toggleClass("col-md-7 col-md-6");
    $('#newcom-search-right').toggleClass("col-md-3 col-md-4");

    $('div.row div.col-md-8 div.row div.col-md-3 div.well').toggleClass("pa-sm");

    $('#div-gpt-placeholder-1, #div-gpt-placeholder-2').toggleClass('center-block');

    // Enlarge column of a parent if it has a child
    var videoDiv = $( "#yourspecialchild" ).parent( ".the-content" ).parent( ".col-md-9" );
    if (videoDiv.length) {
  videoDiv.toggleClass("col-md-9 col-md-12");
  // Move
  var titleDiv = videoDiv.prev();
  if (titleDiv.length) {
    titleDiv.toggleClass("col-md-3 col-md-12");
    titleDiv.insertAfter(videoDiv);
  }
    }

  }

  function setBackgroundMobile() {
    if (typeof backgroundAdMobile300x90 !== 'undefined' && backgroundAdMobile300x90) {
  var ad = jQuery('<a href="' + backgroundAdLink +'" target="_blank"></a>').append(jQuery('<img />',{
    width: 300,
    height: 90,
    src: backgroundAdMobile300x90
  }));

  if (jQuery("#background-ad-mobile").length) {
    jQuery("#background-ad-mobile").append(ad);
  }
  else {
    var subpages = '#wrapper div.row div.col-md-8';
    if (jQuery(subpages).length) {
      var html = jQuery('<div id="background-ad-mobile"></div>').append(ad);
      jQuery(subpages).first().prepend(html);
    }
  }
    }
  }

  $(window).load(function() {
    wallpaperFixViewport();
//  setBackgroundMobile();
    setBackgroundAdPositionX();
    setBackgroundAdPositionY();
    $("#div-gpt-placeholder-0-oop").hide();
  });

  $(window).resize(function() {setBackgroundAdPositionX();});
  $(window).scroll(function() {setBackgroundAdPositionY();});

});

media-w1200-w992.css 

@media (min-width:1200px){
  .col-lg-1, .col-lg-2, .col-lg-3, .col-lg-4, .col-lg-5, .col-lg-6, .col-lg-7, .col-lg-8, .col-lg-9, .col-lg-10, .col-lg-11, .col-lg-12{
    width:auto;
    float:none;
  }

  .container{
    width:990px
  }

  .col-md-1, .col-md-2, .col-md-3, .col-md-4, .col-md-5, .col-md-6, .col-md-7, .col-md-8, .col-md-9, .col-md-10, .col-md-11{
    float:left
  }

  .col-md-12{
    width:100%
  }

  .col-md-11{
    width:91.66666666666666%
  }

  .col-md-10{
    width:83.33333333333334%
  }

  .col-md-9{
    width:75%
  }

  .col-md-8{
    width:66.66666666666666%
  }

  .col-md-7{
    width:58.333333333333336%
  }

  .col-md-6{
    width:50%
  }

  .col-md-5{
    width:41.66666666666667%
  }

  .col-md-4{
    width:33.33333333333333%
  }

  .col-md-3{
    width:25%
  }

  .col-md-2{
    width:16.666666666666664%
  }

  .col-md-1{
    width:8.333333333333332%
  }

  .col-md-pull-12{
    right:100%
  }

  .col-md-pull-11{
    right:91.66666666666666%
  }

  .col-md-pull-10{
    right:83.33333333333334%
  }

  .col-md-pull-9{
    right:75%
  }

  .col-md-pull-8{
    right:66.66666666666666%
  }

  .col-md-pull-7{
    right:58.333333333333336%
  }

  .col-md-pull-6{
    right:50%
  }

  .col-md-pull-5{
    right:41.66666666666667%
  }

  .col-md-pull-4{
    right:33.33333333333333%
  }

  .col-md-pull-3{
    right:25%
  }

  .col-md-pull-2{
    right:16.666666666666664%
  }

  .col-md-pull-1{
    right:8.333333333333332%
  }

  .col-md-push-12{
    left:100%
  }

  .col-md-push-11{
    left:91.66666666666666%
  }

  .col-md-push-10{
    left:83.33333333333334%
  }

  .col-md-push-9{
    left:75%
  }

  .col-md-push-8{
    left:66.66666666666666%
  }

  .col-md-push-7{
    left:58.333333333333336%
  }

  .col-md-push-6{
    left:50%
  }

  .col-md-push-5{
    left:41.66666666666667%
  }

  .col-md-push-4{
    left:33.33333333333333%
  }

  .col-md-push-3{
    left:25%
  }

  .col-md-push-2{
    left:16.666666666666664%
  }

  .col-md-push-1{
    left:8.333333333333332%
  }

  .col-md-offset-12{
    margin-left:100%
  }

  .col-md-offset-11{
    margin-left:91.66666666666666%
  }

  .col-md-offset-10{
    margin-left:83.33333333333334%
  }

  .col-md-offset-9{
    margin-left:75%
  }

  .col-md-offset-8{
    margin-left:66.66666666666666%
  }

  .col-md-offset-7{
    margin-left:58.333333333333336%
  }

  .col-md-offset-6{
    margin-left:50%
  }

  .col-md-offset-5{
    margin-left:41.66666666666667%
  }

  .col-md-offset-4{
    margin-left:33.33333333333333%
  }

  .col-md-offset-3{
    margin-left:25%
  }

  .col-md-offset-2{
    margin-left:16.666666666666664%
  }

  .col-md-offset-1{
    margin-left:8.333333333333332%
  }
}

32.10.3. Interstitial Ad

Interstitial as JW Player is a bit complicated
Gist
(no term)
Load third party code into 1x1 floating ad unit. Insert DFP macro as a third party code first. Refer to dfp:third party code
(no term)

On website 

// Outside of the div
var dfpSlots = {}; // non single request mode

googletag.cmd.push(function() {
// non single request mode
dfpSlots.floating = googletag.defineOutOfPageSlot('/123456/floating', 'interstitial').addService(googletag.pubads()); 

// single request mode
// googletag.defineOutOfPageSlot('/123456/floating', 'interstitial').addService(googletag.pubads());

googletag.pubads().addEventListener('slotRenderEnded', function (event) {
    if (event.slot.getAdUnitPath() === '/123456/floating') {
    document.getElementById('interstitial').style.display = 'none';
    }
});

// ...

googletag.pubads().enableSingleRequest();

// Disable initial load, we will use refresh() to fetch ads.
// Calling this function means that display() calls just
// register the slot as ready, but do not fetch ads for it.
// googletag.pubads().disableInitialLoad();

googletag.enableServices();

});

<div id="interstitial">
  <script type="text/javascript">
googletag.cmd.push(function() { googletag.display("interstitial"); });
  </script>
</div>
(no term)

On DFP, Custom Creative Type postMessage to website to create jQuery UI dialog and later receives a message to create content inside the iframe 

<script type="text/javascript">
  (function() {
'use strict';
var temp = '%%VIEW_URL_UNESC%%'; // Custom Code requires VIEW_URL_UNESC or VIEW_URL_ESC macro..
window.onload = function() {
  function receiveMessage(e) {
    if (e.data == 'load_interstitial_ad') {
      var html = '<div style="width:640px;height:480px;">' +
      '<script src="insert third party code"'+'>'+'</'+'script>'+
      '<noscript>' +
      '<a href="insert_third_party_code" target="_blank"><img src="insert_third_party_code" border="0" width="640" height="480"></a>'+
      '</noscript>'+ 
    '</div>';
      document.write(html);
    }
  }
  window.addEventListener('message', receiveMessage);
  var sendData = {action:"dfp_load_interstitial_ad", width:"640", height:"480"};
  parent.postMessage(sendData,'*');
}
  })();
</script>
(no term)

On website Once received message from iframe, load internal script and only load it once 

var dfpInterstitial = {};
dfpInterstitial.loadCount=0;

function newcomReceiveMessage(e) {
  if (e.data == 'dfp_load_interstitial_ad' && dfpInterstitial.loadCount == 0) {
dfpInterstitial.loadCount++;
dfpInterstitial.width =e.data.width;
dfpInterstitial.height =e.data.height;
// var cssLink = parent.jQuery("<link rel='stylesheet' href='https://ajax.googleapis.com/ajax/libs/jqueryui/1.10.2/themes/ui-lightness/jquery-ui.css'>");
// jQuery('head').append(cssLink);

['//ajax.googleapis.com/ajax/libs/jqueryui/1.10.2/themes/ui-darkness/jquery-ui.css',
 '/wp-content/themes/xxx/css/interstitial.css'].forEach(function (src) {
  var _css = document.createElement("link");
  _css.rel = 'stylesheet';
  _css.type = 'text/css';
  _css.href = src;
  document.head.appendChild(_css);
});

[ 'https://ajax.googleapis.com/ajax/libs/jqueryui/1.10.2/jquery-ui.min.js','/sites/all/themes/abc/js/interstitial.js'
].forEach(function(src) {
  var script = document.createElement('script');
  script.src = src;
  script.async = false;
  document.head.appendChild(script);
});
// var jsLink = jQuery("<script type='text/javascript' src='http://yourwebsite.io/interstitial.js'>");
// jQuery('head').append(jsLink);
  }
}
window.addEventListener('message', newcomReceiveMessage, false);
(no term)

On website, interstitial.js Make ad unit div as a dialog using jQuery UI. jqueryui:dialog jQuery UI Dialog runs script inside the container twice. The script is setup in DFP which only posts or receives message Even though the same message is posted twice but dfpInterstitialCount ensures this js only runs once 

jQuery(function(){
  var targetDialog = '#interstitial'; // TODO
  var iframeId = jQuery(targetDialog).find('iframe').get(0).id;
  var adWidth = dfpInterstitial.width;
  var adHeight = dfpInterstitial.height;

  jQuery(targetDialog).dialog({
width: 'auto',
height: 'auto',
resizable: false,
draggable: false,
closeOnEscape: true,
modal: true,
create: function (event, ui) {
  jQuery("#ui-dialog-title-dialog").hide();
  jQuery(".ui-widget-content").css({"background-color":"transparent","background":"transparent","border":"none"});
  jQuery(".ui-dialog-titlebar").css({"background-color":"transparent","background":"transparent","border":"none"});
  jQuery(".ui-dialog-titlebar").removeClass('ui-widget-header');

  // In Single Request Mode (initial load), every time .display() is called, the div will be refilled with new ad
  // In Non-Single-request mode (after initial load), need to do .refresh()
  if (typeof dfpSlots !== "undefined" && typeof dfpSlots.floating !== "undefined") {
    googletag.cmd.push(function() {
      googletag.pubads().refresh([dfpSlots.floating]);
    });
  }
  jQuery(targetDialog).find('iframe').width(adWidth);
  jQuery(targetDialog).find('iframe').height(adHeight);
},
open: function() {
  jQuery('.ui-widget-overlay').addClass('custom-overlay');
  jQuery(this).closest(".ui-dialog")
    .find(".ui-dialog-titlebar-close")
    .removeClass("ui-dialog-titlebar-close")
    .html("<span class='ui-button-icon-primary ui-icon ui-icon-closethick'></span>");
  jQuery(".ui-dialog-titlebar button").css({"border":"none"});
  jQuery('.ui-widget-overlay').bind('click',function(){
    jQuery(targetDialog).dialog('close');
  })
},
close: function() {
  jQuery('.ui-widget-overlay').removeClass('custom-overlay');
}
  });

  setTimeout(function () {
jQuery(targetDialog).show();
var receiver = document.getElementById(iframeId).contentWindow;
receiver.postMessage('load_interstitial_ad', '*');
  }, 3000);

});

.ui-widget-overlay.custom-overlay {
  background-color: black;
  background-image: none;
  opacity: 0.7;
}

32.11. Pre-roll or Linear Video Ad

  • The ad unit should have a Video (VAST) size e.g. 640x360v
  • In the line item, select the same video size as the inventory size and also select the ad unit
  • Add a new Creative Set and select Linear then Video (not VPAID) for pre-roll, mid-roll or post-roll
  • Upload the video file that a higher resolution e.g. 1920x1080. DFP will create all lower resolution videos
  • Your video creative is setup. You need to generate the tag for the ad unit
  • Use Video Suite Inspector to test the video ad. If the ad unit is newly created, it could take an hour to preview the video ad..
  • Delivery > Troubleshoot > Video simulates the request taking under/over delivery into account
  • DFP Video Tag
    • https://pubads.g.doubleclick.net/gampad/ads?sz=640x360&iu=/{network_id}/{VAST_code_name}&ciu_szs=640x90&impl=s&gdfp_req=1&env=vp&output=xml_vast2&unviewed_position_start=1&url=[referrer_url]&description_url=[description_url]&correlator=[timestamp]
    • In the tag, use JW Player Targeting Macros to add values into these url parameters. Refer to video:jw
      • The url of the page where the ad will appear &url=__page-url__, and other macros &correlator=__timestamp__, &description_url=__referrer__
    • Final DFP tag in JW is https://pubads.g.doubleclick.net/gampad/ads?sz=640x360&iu=/{network_id}/{VAST_code_name}&ciu_szs=640x90&impl=s&gdfp_req=1&env=vp&output=xml_vast2&unviewed_position_start=1&url=__page-url__&description_url=__referrer__&correlator=__timestamp__

    • In JW, specify the div id which the companion ad will be in and on the website 

        <div id="newcomjwplayerbanner" style="width:640px;height:90px;margin-bottom: 10px;"></div>
  <div id="newcomjwplayer">
<script type="application/javascript" src="//content.jwplatform.com/players/{video_id}-{player_id}.js"></script>
  </div>

32.12. Native Ad, Native Styles, Custom Rendering

Custom Rendering can only be used in mobile apps.

Create a native ad format under Delivery > Creatives > Native Styles Associate this native format with an ad unit. A native format is basically a HTML/CSS template with some placeholders which can be later defined when you upload a creative of Native type to a line item. Create a line item with Inventory sizes = Standard and Native format, and target the ad unit. Upload a creative of Native type. Specify the placeholders.

32.13. SafeFrame API

  • Preview creative doesn't enable SafeFrame. The creative has to be deployed
  • When console shows FriendlyFrame, it shows the container is not SafeFrame
  • It's available for 4 types of creative
    custom
    d:on
    third-party
    d:on
    system-defined and user-defined templates
    d:off
  • http://publisherconsole.appspot.com/safeframe/creative-preview.html

  • SafeFrame methods Inside Custom creative 

    <div id="container">
<script>
  function updateInViewPercentage() {
     var text = $sf.ext.inViewPercentage() + '%';
     document.getElementById('percentage').innerHTML = text;
  }

  $sf.ext.register(728, 90, function(status, data) {
    // 728,90 are inital size of the creative
    // register is needed for calling other $sf.ext.xxx
    // Do nothing. This test doesn't make use of this callback.
  });

</script>
<span>
    <strong>$sf.ext.inViewPercentage</strong>
</span>
<button onclick="updateInViewPercentage();">In view pecentage</button>
<div id="percentage"></div>
</div>

32.14. googletag.Service

  • Methods
    addEventListener(eventType, listener)
    refer to google:gam:gpt:event
    getSlots()
  • googletag.pubads() is a googletag.PubAdsService which extends googletag.Service
    getTargetingKeys()
    e.g. ['publisher', 'site', 'section', 'exclusive', 'sponsor']
    getTargeting(key)
    return array e.g. getTargeting('exclusive')

32.15. googletag.Slot

  • Methods
    • addService(service)
    • getAdUnitPath()
    • getSlotElementId()
    • setCollapseEmptyDiv(collapse, opt_collapseBeforeAdFetch)
    • targeting set on Service level is not assigned here. Refer to google:gam:gpt:service
    • same as above

32.16. Events

googletag.events.Event interface
https://developers.google.com/doubleclick-gpt/reference#googletag.events.Event
serviceName
slot
google:gam:gpt:slot
(no term)
googletag.evetns.Event extension for certain events
googletag.events.SlotRenderEndedEvent
fired when the creative code is injected into a slot. Before the creative's resources are fetched. Extra fields:
isEmpty
true if no ad was returned for the slot, false otherwise
googletag.pubads().addEventListener('slotRenderEnded', function (event) {
    if (event.slot.getAdUnitPath() === '/123456/adunit_name') {
    document.getElementById('div-gpt-ad-1403025754774-0-oop').style.display = 'none';
    }
});

32.17. Google Publisher Console

  • To open, append to URL ?googfc=1 (display console after page is loaded) or ?google_console (keyboard to toggle console display) C-F10
  • Or make a bookmark javascript: googletag.openConsole();

32.18. Passback tag

Situation 1

  1. The webpage makes a call to the DFP ad server using the DFP ad tag
  2. DFP ad server returns an ad containing a third-party ad tag
  3. Third-party ad tag calls the third-party ad server for an ad
  4. Third-party ad server doesn't have an eligible ad, so returns a passback ad tag
  5. Passback ad tag makes a call to DFP to serve an ad matching the specified targeting criteria
  6. DFP server returns an ad that matches the passback ad tag targeting criteria

Situation 2

  1. Passback ad tag makes a call to DFP to serve an ad matching the specified targeting criteria
  2. DFP server returns an ad that matches the passback ad tag targeting criteria

32.20. Custom Fields

  • Custom fields can be used in google:gam:query:filter and as dimensions in reports
    • As query filter can only add filter to one field e.g. only filter 1 pair of key-value. Instead, create multiple Custom Fields for query filter
  • can't be changed after set. only one
    • Order
    • Line Item
    • Creative
  • Editable, Read-only, Hidden (API can submit values)
  • can't be changed after set. 30 custom fields to each object type
    • Number
    • select from either Yes or No
    • enter any text value

32.21. Reports

  • Query filter is to filter the request google:gam:query:filter
    • Dimensions and Metrics are also related to the request

For example, a request might have 2 key-value targeting. If you have this query

Filter
none
Dimension
key-value and line item
Metric
impression and click

The request will end up in 2 places

key-value line item Impression Click
a=xyz LI_1 123 5
b=ijk LI_1 123 5

Add filter key-value contains a=xyz

  • Filter out b=ijk

32.21.1. Downloaded impression

Active View eligible impressions
if creative has Active View enabled and the impression is counted with a downloaded pingback
Active View measurable impressions
close to 100% of above. A fail factor may be cross-domain iframe rendering
Active View viewable impressions
>= 50% area is displayed for >= 1 second. In-stream video: 50% for 2 seconds
Ad server impressions
counted after the ad is downloaded in the user's device. Doesn't require the ad content be fully loaded. Exclude from Ad Exchange and AdSense. Takes 30 mins for new ones to be recorded and displayed
Ad server downloaded impressions
Discontinued. Counted after the ad has started to load on the website. Doesn't require the ad content be fully loaded
(no term)
DFP provides a System Query called Downloaded Impressions to compare with the old impression counting
(no term)
Also now a new metric called Request Type to divide:
  • Goolge Publisher Tag
  • Video Tag
  • GPT Simple URL
  • GPT Light
  • Amp Ad Tag

32.22. Check available inventory - Forecasting

  • Works only for Priority:Standard and Priority:Sponsorship (guaranteed line items) line items
  • Priority:Sponsorship line items that are Paused, Draft or with no creatives are also in the competition
  • Delivery > Orders or Delivery > Line items. Do one of the following
    • Create an order or a line item
    • Forecasting
  • Quantity Max Available is only available in Forecasting for Sponsorship line item
  • Forecast for multiple Priority:Sponsorship with total goal more than 100% is not accurate
    • e.g. There're 150k single requests asking for 4 big boxes
      • There're 10 and only 10 identical Priority:Sponsorship line items and each is eligible to show in any 4 big boxes with goal 100%
      • Forecast shows each line item can get 150k impressions since each one is 100% goal. While each one should get 150k * 4 (big boxes) / 10 = 60k
      • Forecasts shows each gets 150k*10% = 15k While each one should get the same 60k as calculated above
      • In short, Priority:Sponsorship line item used in forecast wins every single request and the goal percentage is applied to become the total forecast. Which is wrong. Percentage for each one should be 4 * 10% = 40%

32.23. Benchmarks and Standards

32.24. User role

  • Built-in
    • administrator
    • not Ad Exchange
    • create/manage orders and run reports on orders they create
    • create/manage/approve/cancel orders, edit targeting criteria and run reports on orders/sales/inventory
    • create/edit orders, edit line items, upload creatives and run reports on orders and creatives
    • run reports and evaluate the effectiveness of campaigns through read-only access to all functionality
    • access to Ad Exchange funtionality, but not Ad Manager. Might not be available on your network

33. XML

33.1. XML DOM

Everything is a node nodeType :: 1 - element, 2 - attribute, 3 - text, 4 - comment, 5 - document nodeName :: read-only, text node name is #text, document node name is #document nodeValue :: element node value is undefined

var elements = xmlDoc.getElementByTagName("title"); console.log(elements.length); var element = elements[0]; var elementToRemove = xmlDoc.getElementsByTagName('book')[0]; xmlDoc.documentElement.removeChild(y); element.parentNode.removeChild(element); // remove myself

xde = xmlDoc.documentElement; newNode = xmlDoc.createElement('book'); / create element node newTitle = xmlDoc.createElement('title'); newText = xmlDoc.createTextNode('A Notebook'); / create text node newTitle.appendChild(newText); / add text node to element node newNode.appendChild(newTitle); / add element node to element node

y = xmlDoc.getElementsByTagName('book')[0]; xde.replaceChild(newNode, y); / replace an element node aTextNode.replaceData(0,8,"Easy"); / replace data of text node (from start to 8 in length) // Use nodeValue, it's easier

var attributes = element.attributes; var attribute = element.getAttribute('lang'); var attributeNode = element.getAttributeNode('lang'); attributeNode.nodeValue; attributeNode.nodeValue = "new attribute value"; element.setAttribute('category', 'food');

element.removeAttributeNode(attributeNode); / remove an attribute node element.removeChild("category"); / remove an attribute node by name // remove multiple attribute nodes, you will have to loop all

var x = element.childNodes[0].nodeName;

33.1.1. Navigate

var fc = element.firstChild;
var lc = element.lastChild;
fc.parentNode;
var nc = fc.nextSibling;
nc.previousSibling;

function get_nextSibling(n) {
    // Bypass empty text node while traversing
    var y = n.nextSibling;
    while (y.nodeType != 1) {
    y = y.nextSibling;
    }
    return y;
}

33.2. Do not parse < and &

<![CDATA
]]>

33.3. Comment

<!-- Same as HTML but double -- is not allowed -->

33.4. XML newline is always LF

33.5. XML Element node name can contain -, _ and . but not space

33.6. Attribute value should be HTML entity encoded

Mainly enocde:

  • " => &quot;
  • > => &gt;
  • < => &lt;

33.7. XML namespace

<root 
xmlns:h="http://www.w3.org/TR/html4/"
xmlns:f="http://www.w3schools.com/furniture">

<h:table>
  <h:tr>
    <h:td>Apples</h:td>
    <h:td>Bananas</h:td>
  </h:tr>
</h:table>

<f:table>
  <f:name>African Coffee Table</f:name>
  <f:width>80</f:width>
  <f:length>120</f:length>
</f:table>

</root>

33.8. XSLT

XSLT stands for XSL (Extensible Stylesheet Language) Transformations XSLT uses XPath to navigate through elements.

XSLT code 

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="/">
<!-- Associate the template with the root of the XML source document -->
<!-- A template contains rules to apply when a specified node is match -->
  <h2>My CD Collection</h2>
    <table border="1">
      <tr bgcolor="#9acd32">
        <th style="text-align:left">Title</th>
        <th style="text-align:left">Artist</th>
      </tr>
      <xsl:for-each select="catalog/cd">
<!-- for-each loop, select contains XPath expression -->
<!-- select="catelog/cd[artist='Bob Dylan']" -->
<!-- =, !-, &lt;, &gt; -->
     <xsl:sort select="artist" />
<!-- sort -->
      <tr>
        <td><xsl:value-of select="title"/></td>
        <td><xsl:value-of select="artist"/></td>
        <xsl:if test="price &gt; 10">
        <td><xsl:value-of select="price"/></td>
        </xsl:if>
<!-- choose, when, otherwise -->
       <xsl:choose>
        <xsl:when test="price &lt; 10">
         <td bgcolor=""><xsl:value-of select="price"/></td>
        </xsl:when>
        <!-- Multiple when's -->
        <xsl:otherwise>
         <td><xsl:value-of select="price"/></td>
        </xsl:otherwise>
       </xsl:choose>
<!-- choose, when, otherwise. -->
      </tr>
      </xsl:for-each>
<!-- Close a loop -->
    </table>
</xsl:template>
</xsl:stylesheet>

XML file 

<?xml version="1.0" encoding="UTF-8"?>
<catalog>
  <cd>
    <title>Empire Burlesque</title>
    <artist>Bob Dylan</artist>
    <country>USA</country>
    <company>Columbia</company>
    <price>10.90</price>
    <year>1985</year>
  </cd>
  <cd>...</cd>
  ...
</catalog>

Template and subtemplates 

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="/">
 <html>
 <body>
  <xsl:apply-templates/>
 </body>
 </html>
</xsl:template>

<xsl:template match="cd">
 <p>
 <xsl:apply-templates select="title"/>
 <xsl:apply-templates select="artist"/>
 </p>
</xsl:template>

<xsl:template match="title">
 Title: <xsl:value-of select="."/>
 <br />
</xsl:template>

<xsl:template match="title">
 Artist: <xsl:value-of select="."/>
 <br />
</xsl:template>
</xsl:stylesheet>

33.9. Invalid characters xml:invalid_characters

Characters :: &#x1F; &#x1E;

change to version 1.1 or get rid of the characters 

<?xml version="1.1" encoding="UTF-8" ?>

34. YAML

34.1. Basics

34.2. Sample

---
# three dashes to separate documents
# data types
# - string :: single or double-quoted or no quote at all
# - float
# - integer
# - boolean
#   - True, true, On, on, Yes, yes
#   - False, false, Off, off, No, no
# - null :: `~` or `null` (unquoted)

# Indentation
# - Can be any number of spaces but tabs are not allowed

doe: "a deer, a female deer"
ray: a drop of golden sun
stringWithLineFeed: "this is a line with linefeed \n"
# Use >- or |- to remove linebreak at the end
multipleLinedString: |
  paragraph 1 (keep newlines AKA literal, single newline at the end, AKA clip)
  paragraph 2
onelineStringButTypeInMultipleLines: >
  paragraph 1 (replace newlines with spaces AKA folded)
  paragraph 2
multipleLinedStringWithStrip: |-
  paragraph 1 (strip the trailing newlines)
  paragraph 2


multipleLinedStringWithoutStrip: |+
  paragraph 1 (keep trailing newlines AKS keep)
  paragraph 2


pi: 3.14159
xmas: true
french-hens: 3
calling-birds:
  - huey
  - dewey
  - louie
  - fred
anotherArray: [ 1, 2, 3, 4, 5 ]
anotherArray2: [ "one", "two", "three", "four" ]
anotherArray3: # array of objects
  - type: file
    path: /path/to/file1.log
  - type: file
    path: /path/to/file2.log
xmas-fifth-day: # object or YAML calls it dictionary
  calling-birds: four
  french-hens: 3
  golden-rings: 5
  partridges:
    count: 1
    location: "a pear tree"
  turtle-doves: two
inlineDictionary: { thing1: huey, thing2: louie, thing3: dewey } # in abbreviated form

35. OpenAPI

35.1. Data Types

type and format
https://spec.openapis.org/oas/v3.0.3#data-types

35.1.1. array

  • Unlike general JSON Schema, items keyword is required

  • An array of any type 

    # [ "hello", -2, true, [5.7], {"id": 5} ]
type: array
items: {}

  • An array of 2 types 

    # ["foo", 5, -2, "bar"]
type: array
items:
  oneOf:
    - type: string
    - type: integer

  • An array of an array 

    # [ [1, 2], [3, 4] ]
type: array
items:
  type: array
  items:
    type: integer

  • An array of an object 

    # [ {"id": 5}, {"id": 8} ]
type: array
items:
  type: object
  properties:
    id:
      type: integer

    # Array of Pets
type: array
items:
  $ref: '#/components/schemas/Pet'

  • An array's length 

    # Without minItems, an empty array is valid

type: array
items:
  type: integer
minItems: 1
maxItems: 10

  • Unique items 

    type: array
items:
  type: integer
uniqueItems: true
# [1, 2, 3] – valid
# [1, 1, 3] – not valid
# [ ] – valid

35.1.2. Markdown

35.2. schema - reuse properties

  • $ref in
    • requestBody
    • schema
    • parameters
    • allOf
  • allOf in
    • schema
paths:
  '/customers':
    post:
      tags:
        - Customer
      requestBody:
        $ref: '#/components/requestBodies/CreateCustomer'
      responses:
        201:
          description: Created
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/CustomerId'
    get:
      tags:
        - Customer
      responses:
        200:
          description: OK
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/Customer'
  /customers/{CustomerId}:
    get:
      tags:
        - Customer
      parameters:
        - $ref: '#/components/parameters/CustomerId'
      responses:
        200:
          description: OK
          content:
            application/json:
              schema:
                allOf:
                  - $ref: '#/components/schemas/CustomerProperties'
                  - $ref: '#/components/schemas/CustomerRequiredProperties'
    put:
      tags:
        - Customer
      requestBody:
        $ref: '#/components/requestBodies/CreateCustomer'
      parameters:
        - $ref: '#/components/parameters/CustomerId'
      responses:
        204:
          description: Updated
    patch:
      tags:
        - Customer
      requestBody:
        description: Update customer with properties to be changed
        content:
          application/json:
            schema:
              allOf:
                - $ref: '#/components/schemas/CustomerProperties'
                - type: object
                  properties:
                    Segment:
                      nullable: true
      parameters:
        - $ref: '#/components/parameters/CustomerId'
      responses:
        204:
          description: Updated
components:
  schemas:
    CustomerProperties:
      type: object
      properties:
        FirstName:
          type: string
        LastName:
          type: string
        DOB:
          type: string
          format: date-time
        Segment:
          type: string
          enum:
            - Young
            - MiddleAged
            - Old
            - Moribund
    CustomerRequiredProperties:
      type: object
      required:
        - FirstName
        - LastName
        - DOB
    Customer:
      allOf:
        - $ref: '#/components/schemas/CustomerId'
        - $ref: '#/components/schemas/CustomerProperties'
        - $ref: '#/components/schemas/CustomerRequiredProperties'
    requestBodies:
      CreateCustomer:
        description: Create a new customer
        content:
          application/json:
            schema:
              allOf:
                - $ref: '#/components/schemas/CustomerProperties'
                - $ref: '#/components/schemas/CustomerRequiredProperties'

36. AI

36.1. OpenAI

36.1.1. DALL-E

Image generation
https://openai.com/product/dall-e-2

37. Software Programming

37.1. Programming Paradigm

  • A way to classify programming languages based on their features. A language can have multiple paradigms
    • Which execution model? Behavior of elements of the language
    • e.g. grouping a code into units along with the state that is modified by the code
    • Style of syntax and grammar

37.1.1. Imperative programming

  • Imperative programs spend lines of code describing the specific steps (flow control) used to achieve the desired results: How to do things
    • The steps mutate the program's state

37.1.2. Declarative programming

  • Declarative programs abstract the flow control process, and instead spend lines of code describing the data flow: What to do. The how gets abstracted away
  • To minimize or eliminate side effects
  • Examples
    • Functional Programming
    • SQL
  • In a way, declarative programming is a layer of abstraction on top of imperative programming
  • Core concepts
    • Immutability
    • Separating functions and data
    • functions are treated like any other variable e.g. can be passed as an argument to other functions, returned by another function, be assigned as a value to a variable
    • function that does at least one of the following
      • take one or more functions as arguments
      • return a function as its result
      • To achieve e.g. strategy pattern

    • Partial Application (Currying) 

      $add = fn($x, $y, $z) => $x + $y + $z;
$add_partial = fn($x) => fn($y) => fn($z) => $add($x, $y, $z);
// some code later on
$add_5 = $add_partial(5); // the value for the first parameter is confirmed
// some code later on
$add_5_and_6 = $add_5(6); // the value for the second parameter is confirmed
$total = $add_5_and_6(7); // eq. to $add_partial(5)(6)(7)
      • To achieve procedural operations, or constructor or method injection
    • Recursion
      • As the primary and only means of looping
    • Function Composition
      • Provide other functions to a composed function, running the composed function will run these functions in series e.g. separate first/last names from full name, add initial based on first/last names. Chaining of these functions
      • Functional Programming generally does not have if statements, loop constructs etc.
      • lets us decompose a data structure by its shape instead of its content
      • To achieve mediator pattern

37.1.3. Functional Programming (FP)

  • See Declarative Programming
  • FP is the process of building software by composing pure functions, avoiding shared state, mutable data, and side-effects
  • Functional code tends to be more concise, more predictable, and easier to test than imperative or object oriented code
    • An immutable object is an object that can’t be modified after it’s created. Conversely, a mutable object is any object which can be modified after it’s created.
  • Pure vs impure functions. A pure function must satisfy both of:
    Referential transparency
    the function always gives the same return value for the same arguments. This means that the function cannot depend on any mutable state
  • a(b(c(x)))
  • the function cannnot cause any side effects which may include I/O (console.log), modifyng a mutable object, reassigning a variable, etc.
    • Don't trigger any external process
  • Monad
  • VS OOP
    • FP is better at
      • ensuring stateless code
      • heavy duty data manipulation
      • more terse
    • OOP is better at
      • managing state in general
      • handling interface interactions
      • more verbose
37.1.3.1. Avoid conditions

  • Guard clauses or early returns 

    function abc() {
    if (...) {
        return 'special case #1';
    }
    if (...) {
        return 'special case #2';
    }
    return 'default';
}
37.1.3.2. Avoid reassigning variables

https://blog.sapegin.me/all/avoid-reassigning-variables/

  1. Incremental computations
    const VIDEO_VALIDATIONS = [
    {
    // Must provide either both a height + width, or neither
    isValid: video =>
        validateHeightWidthConsistency(video.videoFiles),
    message: ERROR_MESSAGES.InconsistentWidthHeight
    },
    {
    // Must have ONE OF either a file or a URL
    isValid: video => validateVideoFileAndUrl(video.videoFiles),
    message: ERROR_MESSAGES.InvalidVideoFiles
    },
    {
    // Video URL must be a valid link
    isValid: video => validateVideoURL(video.videoFiles),
    message: ERROR_MESSAGES.InvalidVideoURL
    },
    {
    // Title cannot be blank
    isValid: video => !!video[INPUT_TYPES.Title],
    message: ERROR_MESSAGES.BlankTitle
    },
    {
    // ID must be alphanumeric
    isValid: video =>
        video[INPUT_TYPES.Id].match(ID_PATTERN) !== null,
    message: ERROR_MESSAGES.InvalidId
    }
];

const validateVideo = video => {
    return VIDEO_VALIDATIONS.map(({isValid, message}) =>
    isValid(video) ? undefined : message
    ).filter(Boolean);
};

const printVideoErrors = video => {
    console.log(validateVideo(video).join('\n'));
};
  2. Building complex objects
    const hasDateRange = dateRangeFrom && dateRangeTo;
const queryValues = {
    sortBy: sortField,
    orderDesc: sortDirection === SORT_DESCENDING,
    words: query,
    from: hasDateRange && format(dateRangeFrom.setHours(0, 0, 0, 0), DATE_FORMAT),
    to: hasDateRange && format(dateRangeTo.setHours(23, 59, 59), DATE_FORMAT)
};
  3. Indeterminate loops

    Reassignments aren’t pure evil and exterminating all of them won’t make your code better. They are more like signs: if you see a reassignment, ask yourself if rewriting the code without it would make it more readable. There’s no right or wrong answer, but if you do use a reassignment, isolate it in a small function, where it’s clear what the current value of a variable is.

37.2. Side Effect

An operation, function or expression is said to have a side effect if it modifies some state variable values outside its local environment, that is to say has an observable effect besides returning a value (the main effect) to the invoker of the operation

37.3. DevOps

  • Methodologies
    • People over process over tools
    • Continuous delivery
    • Lean Management
      • Work in small batches
      • Work in progress limits
      • Feedback loops
      • Visualization
    • Visible ops change control
      • Eliminate fragile artifacts
      • Crate a repeatable build process
      • Manage dependencies
      • Create an environment of continous improvement
    • Infrastructure as code
      • Systems treated like code
      • Checked into source control
      • Reviewed, built and tested
      • Managed programmatically
  • 10 Best practices
    • Chaos Monkey
    • Blue/Green Deployment
    • Dependency Injection
    • Andon Cords (anyone can pull to stop the production line)
    • The Cloud (fast API services)
    • Embedded Teams
    • Blameless Postmortems
      • The people whose actions have contributed to an accident can give a detailed account of 5 W's
      • And they can give this detailed account without fear of punishment
      • The detailed account gives an understanding of the mechanism, pathology, and operation of the failure
      • The account also guarantees that it will repeat
      • It's a meeting within 48 hours of the incident
      • Have a third party run it
    • Public Status Page
    • Developers on Call
    • Incident Command System
  • Glossary
    Provision
    Making a server ready for operation, including hardware, OS, system services, network connectivity
    Deployment
    automatically deploying and upgrading applications on a server
    Orchestration
    performing coordinated operations across multiple systems
    Configuration management
    management of change control for system configuration after initial provision; maintaining and upgrading the application and application dependencies
    Imperative/procedural
    commands necessary to produce a desired state are defined and executed
    Declarative/functional
    a desired state is defined, relying on the tool to configure a system to match that state
    Idempotent
    the ability to execute repeatedly, resulting in the same outcome
    Self service
    the ability for an end user to initiate a process without having to go through other people
    Continuous Integration

    is the practice of automatically building and unit testing the entire application frequently. Ideally, every source code check in

    • CI Phases and corresponding tools
    • Version control
    • CI systems (Jenkin, CloudBees, Bamboo, CircleCI, TravisCI)
    • Build (Make/Rake, Maven, Gulp)
    • Test (JUnit, golint/gofmt, RuboCop, Robot, Protractor, Cucumber, Selenium, Sauce Labs, ApacheBench, JMeter, Brakeman, Veracode)
    • Artifact repository (Artifactory, Nexus, Docker Hub, AWS S3)
    • Deployment (Rundeck, UrbanCode, ThoughtWorks, Deployinator)
    • Docker Swarm, Google Kubernetes, Apache Mesos
    Continuous Delivery
    is the addtional practice of deploying every change to to a production like environment and performing automated integration and acceptance testing
    Continuous Deployment
    extends this to where every change goes through full enough automated testing

37.4. Visual Regression Testing

37.4.1. Tools

https://github.com/mojoaxel/awesome-regression-testing https://www.creativebloq.com/features/the-5-best-visual-regression-testing-tools

Headless browser

PhantomJS
WebKit
SlimerJS
Firefox

CasperJS :: JavaScript navigation scripting and testing utility for PhantomJS and SlimerJS ResemberJS :: JavaScript / HTML5 library for making image comparisons.

https://backtrac.io/

BackstopJS 3

37.5. Security Scans

  • Tinfoil Security
  • SiteImprove

37.6. Performance Testing

37.7. Code Quality

37.7.1. Cyclomatic Complexity

  • Number of decision points in a method plus one for the method entry
  • Decision points
    • if, while, for and switch's case and default
    • Not including else, class defintion
  • 1 to 4
  • 5 to 7
  • 8 to 10
  • 11+

37.7.2. TTD: refactor legacy code

  • Add pin-down tests to cover 100% legacy code
  • Refactor
    • Start from small refactoring e.g. convert string, numbers and other hard coded variables to constants
    • De Morgan's Laws
    • Catalog of refactorings
  • usually after refactoring, a new feature needs to be implemented
    • Add small test and implement smallest incremental amount of code to pass the test

37.7.3. Code Climate

37.7.4. Code Scene

  • CodeScene.io identifies hotspot files that generate churns

37.7.5. SonarCloud.io

  • Cloud version of the on-premise version SonarQube
  • Visualize duplication

37.7.6. Snyk

  • Check security for dependencies used in code

37.7.7. De Morgan's Laws

  • !a && !b eq. to !( a || b )
  • !a || !b eq. to !( a && b )

37.7.8. S.O.L.I.D. - Solid Principles

37.7.8.1. Single responsibility :: A class should have one, and only one reason to change
37.7.8.2. Open/closed principle
  • Entities should be open for extension, but closed for modification
  • Open to extension means adding subclasses as needed
  • Closed to modification avoids "tweaking" the code to handle new situations
  • Example
    Before
    rectangle, circle, areaCalculate
    Should be
    interface:areaCalculate, then rectangle and circle implement this interface
  • In short, we should create abstract or parent classes or implement dependency injection or Composition
37.7.8.3. Liskov substitution principle
  • Subclass/derived class should be substitutable for their base/parent class
  • Constrains subclass design
  • Helps programmers design good polymorphism
  • Subclasses shared methods in base/parent class should have the same type of input and output
  • In short, the method signature has to be the same
    • The constructor could be different
37.7.8.4. Interface segregation principle
  • A Client should not be forced to implement an interface that it doesn’t use
  • A client should depend on the smallest set of interface features: the fewest methods and attributes
  • Example
    Before
    WorkerInterface{ work, sleep }, HumanWorker, RobotWorker (Robot implements an empty sleep method)
    Should be
    WorkAbleInterface{ work } , SleepAbleInterface{ sleep }, HumanWorker, RobotWorker
37.7.8.5. Dependency inversion principle (DIP)
  • High-level modules should not depend on low-level modules. Both should depend on abstractions
  • Abstractions should not depend on details. Details should depend on abstractions
  • Just changing the dependency module and High-level module will not be affected by any changes to the Low-level module
  • A direct dependency, on a concrete class, needs to be "inverted", which means to refer to an abstraction
  • Depend on abstraction classes or interfaces
  • Avoid concrete class name dependencies
  • Example
    Before
    MySQLConnection, PasswordReminder{ __construct(MySQLConnection $dbConnection) }
    Should be
    ConnectionInterface, MySQLConnection imp. ConnectionInterface, PasswordReminder{ __construct(ConnectionInterface $dbConnection) }
  • In short, it means when we do dependency injection, use the parent Interface and not a specific type
  1. Inversion of Control (IoC)
    • Instead of the main function (the app) is fully responsible to sequentially instantiate and call methods, the app is now only responsible to do the binding and instantiation of dependencies, the specific methods are called.
  2. Dependence Injection (DI)
    • A software design technique in which the creation and binding of dependencies are done outside of the dependent class
      • Separates the construction and configuration of services from its usage, relieving the dependent class from the former responsibility, decoupling it from its dependencies, and improving reusability and ease to test
    • 3 ways
      Constructor injection
      when dependencies are provided through the dependent class' constructor
      Interface injection
      when dependencies are provided directly into a dependent class method as an argument
      Setter injection
      when dependencies are provided via a public property of the dependent class

37.7.9. General responsibility assignment software principles (GRASP)

37.8. Drupal

DRUPALVM.com devwithlando.io

Send metrics and logs (e.g. Watchdog) to Elasticsearch

Kibana PHP APM (Application Performance Metrics)

37.9. Machine Learning

37.9.1. Supervised vs Unsupervised Learning

Supervised is to label data at first
given sample data (training set) of X/Y pairs (X is input or feature and Y is output or target or true value), now given X', to predict Y \(\hat{y}\)
  • Examples
    Regression
    Both X and Y are a lot or continuous
    First training example
    (x1, y1) = (123, 321)
    ith training example
    (xi, yi) = (321, 321)
    Classification
    Y is finite, X could be a lot e.g. classify known dogs into 5 categories
  • Put new observations into existing criterion categories based on matches with existing data
  • Observations are labeled with (intended) outcome
    • Label may be subjective e.g. good book vs bad book categories
  • then predict outcome category with other variables
  • then apply model to new data
  • Accuracy is important in supervised learning
    • To avoid biase, sampling data should cover all possible data in all categories (diversities)
  • Focus on small categories and edge cases
    • Label could be biased e.g. good book vs bad book. To limit bias, include labels from a wider range of judges
  • multiple people are making a judgment, do they reach the same conclusion?
    Anchoring bias
    first impressions or judgments are strong and resistant to new info
    • Reflected on AI, once a model is developed, new variables for each case may not have as much influence as they should
    Stability bias
    the model makes consistent categorizations even when the outcome has fundamentally shifted
    • e.g. impressions of a celebrity may drastically shift in public perception
    Occam's Razaor
    One should not multiply entities beyond necessity
    (no term)
    The simplest explanation is usually the correct one
  • It's limited to existing variables and categories in original data. Cannot extrapolate
  • K-NN
  • decision tree
  • regression
(no term)
Unsupervised is let the machine to cluster the data
  • Group cases based on similarity of measured attributes without specific criterion
    • Data only comes with inputs x, but not output labels y. Algorithm has to find structure in the data
    • Examples
      Clustering algorithm
      group similar data points together
      Anomaly detection
      find unusual data points
      Dimensionality reduction
      compress data using fewer numbers
  • K-mean clustering e.g. when there are lots of unlabeled data

37.9.2. Causes of classification errors

  • Bayes' Theorem
    True positive
    people should be put into a particular category and what % of those people out of total
    False positive
    people should not be put into a particular category and what % of those people
    (no term)
    The base rate of the outcome

37.9.3. Decision Tree

  • Predictors
    Sky
    sunny, overcast, rain
    Weekend
    weekday, weekend
    Wind
    low, high
  • Outcome
    • Li goes to the beach or not
  • Decision Tree

    • Sky

      Sunny
      2 yes, 2 no, high/low wind, does not matter
      weekday
      0 yes, 2 no
      weekend
      2 yes, 0 no
      Overcast
      1 yes 3 no, weekday or weekend, does not matter
      high wind
      0 yes 2 no
      low wind
      1 yes 1 no
      Rain
      0 yes, 4 no
      (no term)
      no other leaf nodes, high/low wind and weekday/weekend, do not matter

37.9.4. K-nearest neighbor - K-NN

Preparation
classify 1000 known dogs into 5 categories, and creat 2 predictors: hair length and weight
  • Given a unknown dog with a hair and weight, try to classify it
  • Put the unknown dog into a chart with hair and weight, choose a K value of 5, which means find the 5 closest known dogs
  • Let's say 2 of the 5 closest neighbors are Husky, 3 are Shepherd. The prediction for the dog might be Shepherd

37.9.5. K-mean clustering

37.9.6. Naive Bayes

  • Predictors
    • Hair, height, weight
    • Given a dog with those 3 metrics, try to classify the dog into one of the 3 categories Terrier, Hound or Sport

    • Treat each predictor independently e.g. given only the hair color, try to categorize the dog

      • Hair of brown color is 40% Terrier, 10% Hound and 50% Sport. When doing this, ignore other predictors' values
      • Height of 50cm is 20% Terrier, 10% Hound, 70% Sport
      • Weight of 10lb is 10% Terrier, 5% Hound and 85% Sport
      • Add a weighted multiplier for each predictor e.g. 3 for Hair, 2 for Height and 1 for Weight
      • Hence, a dog with those 3 metrics, being a
      • Terrier is 40%*3 + 20%*2 + 10%*1 = 170%
      • Hound is 10%*3 + 10%*2 + 5%*1 = 55%
      • Sport is 50%*3 + 70%*2 + 85%*1 = 375%
      • Hence, the prediction is the dog is a Sport dog
  • Good for binary or multiclass classification

37.9.7. Bias vs Variance

  • Bias is the gap between the predicted value and the actual outcome
  • Variance is when the predicted values are scattered all over the place
  • High bias and low variance
    • consistently wrong
  • High bias and high variance
    • consistenly wrong in an inconsistent way

37.9.8. Signal vs Noise, underfitting and overfitting

  • Signal is something that can be used to make accurate predictions
  • Noise is natural variances in the data that might not offer any insights
  • When the model has noise predictors, it includes other unnecessary factors, and it is called overfitting
  • Underfitting is some signal predictors are not and should be included in the model

37.10. Ansible

37.11. Licensing software:license

  • The software code that can be viewed and editted is subject of copyright
  • What the software does when it executes is the subject of patent
  • License copyrights, trademarks and patents

37.11.1. Open Source

  • https://opensource.org/ Open-Source Initiative Criteria
    • Free redistribution
    • Must provide source code or make it available
    • Must allow derived works
    • The itegrity of author's code must be maintained
    • No discrimination against persons or groups
    • No discrimination against fields of endeavor
    • Distribution of only one license required to secure rights
    • License must not be specific to product
    • License must not restrict other software
    • License must be technically neutral
  • try to affiliate with an open-source foundation like Apache, Mozilla or Outercurve
    • Create a business entity for succession plan
  • BSD (Berkeley Software Distribution)
  • GPL and other GNU licenses (General Public License)
    All GNU licenses
    http://www.gnu.org/licenses/
    GPL
    http://www.gnu.org/licenses/#GPL
  • MIT license
    • Permissive
    • If you don't care about restrictions or what happens downstream
  • Apache
    • If you want to encourage commercial adoption of your code
    • Only it has an implicit software:license:cla
  • Copyright vs Copyleft
    Copyright
    I give right to someone
    Copyleft

    I give right to someone so that he/she has the same right

    • The same license as the original code must be maintained
    • GPL v3
    • Combining GPL covered source code with non-GPL code does make target code subject to GPL
      Weak Copyleft
      LGPL v3
    • Distribution of binary file libraries (DLLs) does not affect the license of target code that references DLLs
  • Permissive
    • Few restrictions and obligations. Non-Copyleft. Software covered by permissive licenses can be included in copyleft covered software but not the other way around
    • e.g. BSD, MIT, Apache

37.11.2. Creative Commons

https://creativecommons.org/
for content (non-software), embrace copyleft principles
(no term)
Concepts
Attribution
author, creator
(no term)
Restrictions
  • Share Alike
  • Noncommercial
  • No Derivatives

37.11.3. Contributor License Agreements (CLA) software:license:cla

38. Tools

38.1. MacOS

38.1.1. Versions

Catalina
10.15

38.1.2. Escape key not working

  • Activity Monitor > Find Siri > Force Quit > Start again

38.1.3. Screenshot and Screen Recording ⇧ ⌘ 5

  • Recording
    Stop
    ⌃ ⌘ Esc
    Options
    turn on microphone
  • Cmd-S-4
    Copy portion of screen
    Cmd-C-S-4
  • Cmd-S-3
    Copy entire screen
    Cmd-C-S-3

38.1.4. Directories

  • ~/Library
    LaunchAgents or LaunchDaemons
    for scheduled or background tasks
    (no term)
    Preferences or /Library/Preferences

38.1.5. Shortcuts

  • https://support.apple.com/en-ca/HT201236
  • System Preferences > Keyboard > Shortcuts
    • To disable
      • Services
    • ⌘ ⇧ a
      • Mission Control
    • F6
      Quick Note
      Fn-q
    • To add
      • App Shortcuts
    • Google Chrome.app
      Move Tab to New Window (added)
      ⇧ ⌘ w
  • ⌥ Brightness Up or Down
  • ⌥ Volume up or down
  • Menu bar
    Ventura
    ⌃ F1 (enable keyboard access) then ⌃ F2
    Monterey
    Fn-m
  • Fn-a
  • Fn-h
  • Fn-n
  • Fn-c
  • Fn-d
  • Fn-e
  • Fn-f
  • Fn-q
  • ⌥ ⌘ Esc
38.1.5.1. Finder
Jump to Home folder of the current macOS user account
⇧ ⌘ h
Create a new folder
⇧ ⌘ n
View the Finder items as icons
⌘ 1
View the Finder items as list
⌘ 2
Use Quick Look to preview the selected items
⌘ y
Open the parent folder
⌘ Up
Show hidden items
⇧ ⌘ .
Go to
⇧ ⌘ g
Go to next tab
⌃ Tab
Permanent delete
⌘ ⌥ Delete followed by ⌘ d to confirm
Search files
⌘ f
Saved Search
.savedSearch is XML
  • ... icon on the top to Show Search Criteria, change view mode and grouping, then Save
(no term)
Tags
  • Caveat
    • For mounted image file
      • Both system and custom tags cannot be searched but can be displayed using Group by tags
      • Custom tags cannot be searched and displayed using Group by tags for Saved Searches
        • Custom tags with a color can be displayed using Group by tags, but only the color names are displayed
Kind
It's not the file extension. Get info and look for Kind
Name
ends with for file extension. Case-insensitive
Raw Query
regex
38.1.5.2. Window
Command + `
switch in-app windows e.g. chrome normal and chrome incognito
Ctrl + Tab
switch in-app tab e.g. chrome tabs
Close current window
Cmd-w
Close all windows of the app
Cmd-M-w
(no term)
No shortcut for maximize current window
Switch to next window of the app
Cmd-`
Hide current app
Cmd-h and bring it back Cmd-Tab
Hide all windows of other apps
Cmd-M-h
Bring back minimized app
Cmd-Tab to the app, before releasing Cmd, press Alt
38.1.5.3. Input
Emoji
Command + Control + Space
fn
switch between Input Source
Go to file top
Command + Fn + Left or Command + Up
Go to file bottom
Command + Fn + Right or Command + Down
Move to previous / next word
Option + Left / Right
Move to beginning / end of a line
Command + Left / Right
Page Down / Up
Fn + Down / Up

38.1.6. Homebrew

38.1.6.1. Install :: https://brew.sh/
38.1.6.2. Install brew version of zsh
# Current Login Shell
# System Pref > Users & Groups > Unlock to make changes > right click on your name > Advanced Options > Login shell
# default is /bin/zsh that is also what comes with MacOS
dscl . -read /Users/$USER UserShell

# should be
# UserShell: /bin/zsh

brew install zsh

which zsh

# should be
# /opt/homebrew/bin/zsh -> ../Cellar/zsh/5.8_1/bin/zsh

# Change the current user to use this as the Login Shell
sudo dscl . -create /Users/$USER UserShell /opt/homebrew/bin/zsh

# restart Terminal, verify the new shell is /opt/homebrew/bin/zsh
dscl . -read /Users/$USER UserShell

# should be /opt/homebrew/bin/zsh
echo $SHELL

# should be -zsh
echo $0

# done!
38.1.6.3. Basics
# like sudo apt-get update, it also updates brew and cask
brew update

# list installed formulas/formulae. A formula is a package definition or a package.
brew list

# show an installed package's installation path
brew list git

# list installed applications
brew list --cask

# see what versions you have, and later switch to a different version
brew list --versions git

# Change versions
brew switch git 2.5.0

# install an application using dmg
brew install --cask firefox

# upgrade a package/formula
brew upgrade git

# upgrade all packages/formula
brew upgrade

# List all added/tapped repositories. Third-Party Repositories = Tap
brew tap

# brew folder/directory which is /opt/homebrew
brew --repository

# Clones `https://github.com/<user>/homebrew-<repo>` to /opt/homebrew/Library/Taps, later formulae will be updated when `brew update` is run
brew tap <user/repo>

# Remove the tap

# Check a package versions, caveats, etc. May do it before installation
brew info git
38.1.6.4. KeyCastr

https://github.com/keycastr/keycastr#readme 

brew install --cask keycastr
38.1.6.5. telnet
brew install telnet
telnet towel.blinkenlights.nl
38.1.6.6. mutagen
brew tap mutagen-io/mutagen
brew install mutagen-io/mutagen/mutagen
brew install mutagen-io/mutagen/mutagen-compose

# /opt/homebrew/Cellar/mutagen/0.13.1
38.1.6.7. svn
brew install svn

svntools have been installed to:
  /opt/homebrew/opt/subversion/libexec

The perl bindings are located in various subdirectories of:
  /opt/homebrew/opt/subversion/lib/perl5

You may need to link the Java bindings into the Java Extensions folder:
  sudo mkdir -p /Library/Java/Extensions
  sudo ln -s /opt/homebrew/lib/libsvnjavahl-1.dylib /Library/Java/Extensions/libsvnjavahl-1.dylib
==> Summary
🍺  /opt/homebrew/Cellar/subversion/1.14.1_4: 234 files, 33.1MB
38.1.6.8. colordiff

brew install colordiff

38.1.6.9. Font
  1. System Fonts
  2. source code pro
    brew tap homebrew/cask-fonts
brew install --cask font-source-code-pro

38.1.7. Display

3024 × 1964
MacBook Pro (14-inch, 2021, M1 Pro)
window.screen.width
1512
window.screen.height
982
window.screen.devicePixelRatio
2

38.1.8. Terminal

open afile, open adir, open ., open ..
open in Finder
open -a TextEdit afile.txt
open a file using an app
ls -al | open -f
redirect stdout to TextEdit
cat a.tsv | open -fa Numbers
redirect stdout to an app
(no term)
Clipboard
  • ls -la | pbcopy, pbcopy < afile.txt
  • pbpaste > afile.txt

  • 4 different clipboards! 

    echo "first" | pbcopy -pboard general
echo "second" | pbcopy -pboard find

pbpast -pboard find
(no term)

Screen capture 

screencapture /path/to/screencapture.png

# interactive
screencapture -i /path/to/screencapture.png

# -m main monitor only. Default all monitors
# -t, format: png, pdf, jpg, tiff
# -C, show cursor
# -T, delay in seconds
# -P, open file with Preview
  • Cmd-S-3
  • interactive
(no term)
sudo shutdown
  • Options
    h
    halt
    r
    reboot
    s
    sleep
  • Time
    now
    sudo shutdown -h now
    (no term)
    +minutes
    (no term)
    yymmddhhmm

38.1.9. Safari

Super refresh / hard refresh
⌥ ⌘ r
Chrome and Firefox
⇧ ⌘ r
Web Inspector
⌥ ⌘ i

38.1.10. NFS

  • Give Full Disk Access to nfsd
    • System Preferences > Security & Privacy > Privacy > on the left Full Disk Access, add another
    • Cmd-S-g then type /sbin/nfsd
sudo nano /etc/exports

# where $U is `id -u`, $G is `id -g`

# Since Catalina, /Users are in another volume
# instead of doing this:
# /Users -alldirs -mapall=$U:$G localhost

# now insert the following
# /System/Volumes/Data -alldirs -mapall=$U:$G localhost

# more about /etc/exports file
# http://scottlab.ucsc.edu/xtal/wiki/index.php//etc/exports

sudo nano /etc/nfs.conf

# insert
# nfs.server.mount.require_resv_port = 0
# more about /etc/nfs.conf file
# http://scottlab.ucsc.edu/xtal/wiki/index.php//etc/nfs.conf

sudo nfsd restart

38.1.11. DNS 

sudo nano /etc/hosts

# clear DNS cache
sudo killall -HUP mDNSResponder

38.1.12. Photos app

  • Repair a photo library
    • In /Applications folder, ⌥ ⌘ double click Photos.app
    • Shared albums will be re-downloaded
  • Create and open a photo library and set it to the System Photo Library
    • In /Applications folder, ⌥ double click Photos.app

38.1.13. Time Machine

  • Disk Utility > Select all Devices and select the external hard drive > Erase
    Format
    APFS
    • See 1
    Scheme / Partition
    GUID Partition Map
    • See 3
    (no term)
    Go to the volume > Click Partition
    • Volume #1
      Format
      Mac OS Extended (case-sensitive and Journaled)
      Name
      Time Machine
    • Volume #2
      Format
      ExFAT
      Name
      PC and Mac
  • Eject external drive fail
    Go to Finder then force quit
    ⌘ ⌥ Escape
    (no term)
    Or logout Mac and login back

38.1.14. Disk Utility

38.1.14.1. Create a blank image
  • Format
    APFS or APFS (Case-sensitive)
    For the latest Mac that uses SSD
    Mac OS Extended (Journaled)
    For macOS 10.12 or earlier
    FAT or ExFAT
    For Windows. For less than 32 GB, use FAT
  • assign a passphrase to access the created image file
  • Partition (aka Scheme / Partition Map)
    GUID Partition Map (GPT)
    All Intel-based and Apple silicon Mac computers. New Microsoft Windows-based computers can use this scheme
    Master Boot Record
    For all Microsoft Windows-based computers
    Apple Partition Map
    For old PowerPC-based Mac computers
  • Image Format
    Sparse Bundle Disk Image
    .sparsebundle file extension
    • Creates / has a lot of files
    • Use Sparse Bundle for macOS over 10.5
    Sparse Image
    .sparsebundle file extension
    • For earlier version of macOS
    Read/write disk image
    .dmg file extension
    • Change the .dmg capacity size later
      • Inject the disk then Disk Utility app > Images > Resize
        • The app can't resize .dmg in APFS format, use command line. Inject disk before running
          • hdiutil resize -size 25G /PATH/TO/DISK/IMAGE.dmg
            • Mount it and you will see the Capacity size is set
            • Expand the Container and the volume to verify the size matches

38.1.15. Other Apps

38.1.15.1. XnViewMP
  • ~/.config/xnviewmp directory is created to store settings (xnview.ini), catalog (XnView.db) and thumbnails (Thumb.db)

38.1.16. Logitech MX Master 3S

# reboot MacBook after running
sudo defaults write /Library/Preferences/com.apple.airport.bt.plist bluetoothCoexMgmt Hybrid

# to revert
sudo defaults delete /Library/Preferences/com.apple.airport.bt.plist bluetoothCoexMgmt

38.2. PhpStorm

38.2.1. Installation and 64 bit

  • Just install the new version and it will guide you to uninstall and keep the settings.
  • For activation, choose License server, address is http://idea.lanyus.com/
    Used before
    https://www.cnblogs.com/oucbl/p/11664610.html, http://idea.qinxi1992.cn/
    From 2019.1.2
    Just 0.0.0.0 https://account.jetbrains.com:443 in /etc/hosts and linux:dns:flush
    Before
    0.0.0.0 account.jetbrains.com 0.0.0.0 www.jetbrains.com in /etc/hosts
  • Help > Check for Update > Change setting to don't auto check update
    • Still can update plugins manually in this way
  • Install Java SE Development Kit (JDK) 64 bit

    • Open C:\Program Files (x86)\JetBrains\PhpStorm 10.0.3\bin\PhpStorm64.exe, Help > About, you should see 

      JRE: amd64
JVM: 64-bit

      If not, set environment variable JAVA_HOME to C:\Program Files\Java\jdk1.8.0_101 and restart PhpStorm64.exe

    • Restart PhpStorm64.exe and open multiple projects to test RAM
    • Finally, edit the shortcut to use PhpStorm64.exe
  • Open a file from command line

    • MacOS

      • Create ~/phpstorm
      #!/bin/sh

open -na "PhpStorm.app" --args "$@"

      chmod ~/phpstorm
sudo ln -s ~/phpstorm /usr/local/bin/phpstorm
  • Installation directories
    • ~/Library/Application Support/JetBrains/PhpStorm2022.3
    • Cache and history files
      • ~/Library/Caches/JetBrains/PhpStorm2022.3
    • Plugins
      • ~/Library/Application Support/JetBrains/PhpStorm2022.3/plugins
    • Logs
      • ~/Library/Logs/JetBrains/PhpStorm2022.3

38.2.2. Material Theme UI

38.2.3. Change config directories

38.2.4. Open big file

Ensure phpStorm is running in 64bit.

https://intellij-support.jetbrains.com/hc/en-us/articles/206544519 This is called idea.config.path Create a custom idea.properties file in one these locations. This is called

For Windows
in %USERPROFILE%\.IntelliJIdeaXX or %USERPROFILE%\.IdeaICXX,
For Windows
actually %USERPROFILE%\.PhpStorm2017.1\config
For *NIX
in ~/.IntelliJIdeaXX or ~/.IdeaICXX
For macOS
in ~/Library/Preferences/IntelliJIdeaXX or ~/Library/Preferences/IdeaICXX

Don't change the system idea.properties at install-path/bin

Help > Edit Custom Properties to create it and edit 

# Changing this will affect global search performance!
# in kb, it's 500mb
# Provide code assistance
idea.max.intellisense.filesize=500000
# Open file size
idea.max.content.load.filesize=500000

Help > Edit Custom VM Options 

-Xmx8192m

# old :: change ~-Xms~ and ~-Xmx~ to ~-Xms2g~ and ~-Xmx1g~

More settings :: http://www.jetbrains.com/help/idea/tuning-the-ide.html

38.2.4.1. Plugin: JB SDK Bintray Downloader

Install JetBrains Plugin Find Action > Get JB SDK Select the most recent version to download and then install: e.g. jbsdk8u152b941_windows_x64.tar.gz %USERPROFILE%\.PhpStorm2017.1\config\jdks\ holds JB SDK libraries downloaded using the plugin

About should show JRE: 1.8.0_152-release-b941 amd64 JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o

Find Action > Switch IDE Boot SDK to switch between installed runtimes. Always use the JetBrains version.

38.2.5. Deployment

  • https://www.jetbrains.com/help/phpstorm/tutorial-deployment-in-product.html
  • Tools > Deployment > Configuration
    • For local deployment, use In place type, and Web server root URL is the file path to the project root directory e.g. file:///E:/li/cssSandbox/, Mappings > Web path on server is /

    • Rsync setting

      • Default for MacOS
      Rsync executable path
      rsync
      Rsync options
      -zar
      Shell executable path
      ssh
      • Rsync using a non-root but sudoer SSH credentials
      (no term)
      See 15.10.23.4
      Rsync executable path
      rsync
      (no term)

      Rsync options 

      -zar
--no-p
--no-g
--no-o
--rsync-path=sudo rsync
      Shell executable path
      ssh
  • ⌥ ⇧ ⌘ x

38.2.6. Search, Find

38.2.6.1. Find Usages ⌥ F7
  • When a variable is a whole array, Find Usages works perfectly. If the variable is $a['abc'], Find Usages won't show functions that change $a['abc'] (not the whole array $a) as Value write but show as under Value read. $a['abc'][] = 'etc'; is always shown as Value write
38.2.6.2. Find file
Search Everywhere
double tap S
Recent Files
C-e or ⌘ e
Switcher
⇧ tab
d
switch to database view
Navigate > File (Go to file)
C-S-r
Find directory
name/ or name\
(no term)
Find file
CamelHumps
sap matches StrangeAnimalPage.html
snake_case
s a p matches strange_animal_page.html
Files in nested directories
a/a/s a p matches calendar\Animals\Animaux\StrangeAnimalPage.html and strange_animal_page.html in the same folder
(no term)
e.g. for Drupal, s/a/m/custom/*.md matches sites/all/modules/custom/*.md files
38.2.6.3. Regex Find
  • Find functions with names that contain _node_
    • function[\s+].*_node
  • Functions with names that end with _node_
    • function[\s+].*_node\(
  • Function calls with specific values for optional arguments
    • Not 100% correct but close enough
    • e.g. PHP function currency_format( $val, $digits=2, $bracket=0, $omitzero=0 )
    • Want to see all function calls with $bracket = 1
    • currency_format\(.*,\s*2\s*,\s*1\s*
    • currency_format\([^\)]*,[^\)]*,\s*2\s*
  • Multiple lines
    • :PROPERTIES:\n:ID:.*\n:END:\n
38.2.6.4. Search Structurally

38.2.7. Language Specifics

38.2.7.1. HTML: Show Applied Styles

Right click on element and select Show Applied Sytles for Tag

38.2.7.2. CSS: vendor prefix

e.g. all vendor prefixes for box-sizing. Type -box-sizing and hit tab

38.2.7.3. JavaScript version
  • Settings > Languages & Frameworks > JavaScript > JavaScript language version to ECMAScript 6
  • For Scratch, right click on the file in Edit window and change language to ES6

38.2.8. View

38.2.8.1. Appearance View > Appearance
View modes
e.g. Zen mode C-S-F12 View > Appearance > Enter Zen Mode
  • Hide all active tool windows and leave the Editor Tabs open. Can toggle
(no term)
Tool Window Bars
  • appears on left of Editor Tabs
(no term)
Status Bar
Status Bar Widgets
select all items to display
(no term)
Navigation Bar M-Home
(no term)
Details in Tree View
  • I choose off
  • In Project, show file dates and size
  • In Database, show description for databases and tables
38.2.8.2. Quick Switch Scheme, Switch Theme C-`, View > Quick Switch Schme
  • Editor Color Scheme
  • Code Style Scheme
    default
    for all projects (default)
    Project
    for current project
  • I choose Eclipse to be my default keymap
  • presentation mode, fullscreen mode, zen mode
  • Theme
  • Material Theme presets
38.2.8.3. Parameter Info
  • Inside the function parameter abc(placeCursorHere)
  • 1.1
38.2.8.4. Quick Definition C-S-i

38.2.9. XPath

Make the xml file part of a project to use Evaluate XPath… (right click)

38.2.10. Navigate, Display, Select inside Editor

https://www.jetbrains.com/help/phpstorm/navigating-through-the-source-code.html#find_cursor_edit

38.2.10.1. Edit > Paste > Paste from history (access multiple clipboards) :: C-S-v ⌘ ⇧ v
38.2.10.2. Edit > Find
Replace
⌘ r
  1. Search Structurally
    1. Static method calls of a class
      <searchConfiguration name="Utils::debugDump" text="$a$::debugDump($c$)" recursive="true" type="PHP" search_injected="false">
  <constraint name="__context__" within="" contains="" />
  <constraint name="a" nameOfExprType="BrokerToMAUtils" within="" contains="" />
  <constraint name="c" minCount="3" maxCount="2147483647" within="" contains="" />
</searchConfiguration>
    2. Static method calls of a class using regex
      • The 3rd parameter not starting with 0 or whitespace
        • When ^ is used in a regex, better to use $ to match the whole 3rd parameter
      <searchConfiguration name="Utils::debugDump" text="$a$::debugDump($b$, $c$, $d$, $e$)" recursive="true" type="PHP" search_injected="false">
  <constraint name="__context__" within="" contains="" />
  <constraint name="a" nameOfExprType="BrokerToMAUtils" within="" contains="" />
  <constraint name="c" within="" contains="" />
  <constraint name="b" within="" contains="" />
  <constraint name="d" regexp="^[^0\s].*$" within="" contains="" />
  <constraint name="e" minCount="0" maxCount="2147483647" within="" contains="" />
</searchConfiguration>
    3. Static method calls of all child classes of a specific parent class

      • First, create a search template BasicEnumChildClasses that finds all child classes of a parent class called BasicEnum abstract class $BasicEnumChildClass$ extends $b$ {} 

        <searchConfiguration name="BasicEnumChildClasses" text="abstract class $BasicEnumChildClass$ extends $b$ {}" recursive="true" type="PHP" case_sensitive="true">
  <constraint name="__context__" within="" contains="" />
  <constraint name="b" regexp="BasicEnum" within="" contains="" />
  <constraint name="BasicEnumChildClass" target="true" within="" contains="" />
</searchConfiguration>

      • Create another search template BasicEnumChildClass Static Method Calls which reuses $BasicEnumChildClass$ from the previous template $BasicEnumChildClass$::$b$() 

        <searchConfiguration name="BasicEnumChildClass Static Method Calls" text="$BasicEnumChildClass$::$b$()" recursive="true" type="PHP" case_sensitive="true">
  <constraint name="__context__" within="" contains="" />
  <constraint name="b" within="" contains="" />
  <constraint name="BasicEnumChildClass" reference="BasicEnumChildClasses" within="" contains="" />
</searchConfiguration>
    4. Function Calls
      sql_fetch_assoc($a$($p$), $pp$)

      sql_fetch_assoc followed by function calls of sql_query or sql_prep 

      <searchConfiguration name="sql_fetch_assoc($a$($p$), $pp$)" uuid="d8bdd089-5864-3ad5-8649-fad285ed10f8" text="sql_fetch_assoc($a$($p$), $pp$)" recursive="true" type="PHP" case_sensitive="true">
  <constraint name="__context__" within="" contains="" />
  <constraint name="a" regexp="sql_prep|sql_query" within="" contains="" />
  <constraint name="p" minCount="0" maxCount="2147483647" within="" contains="" />
  <constraint name="pp" minCount="0" maxCount="2147483647" within="" contains="" />
</searchConfiguration>
    5. My Search Templates

      https://www.jetbrains.com/help/phpstorm/2022.2/search-templates.html

      PHP 

      return $a$;
38.2.10.3. Edit > Find Usages
Find Usages
C-g or ⌥ F7
  • If a JavaScript method is called inside a PHP Heredoc, you'll need to open the Edit Fragment, then you can Find Usages at the place where the JS method is defined
    • You can still click on the called method insdie PHP Heredoc to locate the JS method declaration place
  • TS: Control flow is too big to analyze
    • When a file has too many lines inside a block scope e.g. switch with many lines
    • or a file has a lot of function/method calls inside a block scope
    • The inspection (e.g. Find Usages, Autocompletion) is turned off for that file
Next highlighted usage
⌃ ⌥ Down / Up
  • Just need to hover on the variable and hit to see the next usage
Highlight usages in File
⇧ ⌘ F7
Find Usages in File
Edit > Find Usages > Find Usages in File ⌥ ⌘ F7
(no term)
See PHPStorm Structural Search
(no term)
Find Usages Settings
Scope
default Project Files and Libraries
(no term)
General
Search for text occurrences
default unchecked
Skip results tab with one usage
default unchecked
38.2.10.4. Show Whitespaces - Show unprintable characters (TAB)

Find Action > search Show Whitespaces

38.2.10.5. Multiple cursors, move cursor, selection, navigate cursor
Create a cursor at caret
M-click or
  • Hit Ctrl / ⌥ twice and hold Ctrl / ⌥, then press an arrow key to create a cursor
  • Clone Caret Below or Clone Caret Above when C-S-a (no shortcuts defined in Eclipse)
(no term)
Select and make new cursors
Add selection for next occurance
M-y
Select all occurances
C-M-y
Reduce to only one cursor
Esc
38.2.10.6. Extend selection :: S-M-Up S-M-Down ⌥ Up
38.2.10.7. Move Caret to Line End :: End ⌘ Right ⌃ e
38.2.10.8. Move Caret to Line Start :: Home ⌘ Left ⌃ a
38.2.10.9. Move Caret to Code Block Start or end (open bracket, starting bracket) :: C-[, C-], ⌥ ⌘ [
  • Repeat to keep going up to parent scope code blocks
38.2.10.10. Move Caret to Matching Brace :: C-S-p ⌃M
  • First hit goes to the start brace, hit again to go to the end brace
  • ( [ < " '
38.2.10.11. Move Caret to Text Start (top of file) :: ⌘ Home
38.2.10.12. Navigate > Class C-S-t, File C-S-r, Symbol C-M-S-n
38.2.10.13. Navigate > Jump to navigation bar ⌘ up
38.2.10.14. Navigate > File Structure (popup window) :: C-F3 ⌘ F12
38.2.10.15. Navigate > Type Hierarchy F4, Method Hierarchy, Call Hierarchy
Type
Class inheritance hierarchy
Method
Hierarchy of a method of a class
  • Defined in this class, superclasses? do subclasses also define this method?
(no term)
Call
  • Sometimes Expand All to get callers and callees will end up in infinite loop
  • Caller Hierarchy
    • Which functions call this method/class? And which functions call those functions?
  • Callee Hierarchy
    • Which functions get called by this method/class? And which functions also call those functions?
38.2.10.16. Navigate > Implementation(s) C-t ⌥ ⌘ B
38.2.10.17. Navigate > Super Method (opposite of Navigagte > Implementaion(s)) :: ⌘ U
38.2.10.18. Navigate > Declaration and Usages F3 or C-leftClick or ⌘ + b
38.2.10.19. Navigate > Select in (select opened file in other window) ⌥ + F1
38.2.10.20. Navigate > Column (Go to line) ⌘ + L
38.2.10.21. Navigate > Back (move to previous place on editor) :: ⌘ [
38.2.10.22. Navigate > Last edit location :: ⇧ + ⌘ + Backspace
38.2.10.23. Navigate > Next Highlighted Error (next error in an Editor tab) :: S-F1 Mac: F2 ⇧ F2
38.2.10.24. Navigate > Navigate in File > Next Change, Previous Change (Git) :: ⇧ ⌃ ⌥ Up/Down
38.2.10.25. Navigate > Navigate in File > Previous method / Next method :: C-S-Up, C-S-Down, ⇧ ⌃ Up/Down
  • Move to previous sibling element in HTML. If there's none, then move to the start of the parent element
38.2.10.26. View > Recent Locations C-S-e M-left or right
38.2.10.27. View > Parameter Info (function parameters) :: ⌘ P, ~~
38.2.10.28. Error Description :: ⌘ F1
38.2.10.29. Scroll to Centre ⌃ l
38.2.10.30. Custom code folding regions
  • Select the block of code you want to fold
  • C-M-t or surround with
  • Choose either NetBeans style or VisualStudio style. Don't mix 2 styles
38.2.10.31. Window > Editor Tabs - Split and multiple Editor Tabs
  • 3 or 2
    Split right
    ⇧ enter
    Unsplit all
    no shortcut
  • Window > Editor Tabs
    Goto Next Splitter
    ⌥ tab
    Select Next Tab
    ⇧ ⌘ [
    (no term)
    Split Vertically
    (no term)
    Change Split Orientation
    Maximize Editor
    The opposite is Normalize Splits
38.2.10.32. Window > Active Tool Window > Maximize Tool Window ⇧ ⌘ '
38.2.10.33. Window > Active Tool Window > Hide all windows :: ⇧ + ⌘ + F12
required
Windows > Store Current Layout as Default
38.2.10.34. Window > Next project window ⌘ ⌥ `
38.2.10.35. Open source in new window (open current file in new window) :: S-F4 ⇧ F4
38.2.10.36. Close tab C-F4
38.2.10.37. Show context menu ⌥ + Enter
38.2.10.38. Quick Definition :: C-S-i ⌥ Spacebar
38.2.10.39. Quick Documentation (has browser support for CSS) :: C-S-Space F1
38.2.10.40. Goto by Reference Actions - File Path ⌥ ⌘ F12
38.2.10.41. Goto by Reference Actions - Jump to Navigation Bar :: ⌘ Up
38.2.10.42. VCS Operations ⌃ + v

38.2.11. Difference Viewer

⌃ + ⌘ + D
Toggle viewer control options e.g. Ignore options, Highlight, collapse unchanged parts
⌃ + ⇧ + Tab
Switch between left and right pane
F7 or ⇧+F7
Next / previous difference
⇧ + ⌘ + [
Compare next / previous file
⌘ + Down
Jump to source file
(no term)
Highlight
  • Highlight split changes

38.2.12. Project View / Project Tree ⌘ 1

38.2.12.1. 3 dots > Always Select Opened File
38.2.12.2. Collapse All ⌘ -
38.2.12.3. Rename Project
  • From Project View, select the root folder of the current project, and go to File > Rename project

38.2.13. Git ⌘ 9

  • Branches > right click a branch e.g. master and the current branch is release
    • Show Diff with Working Tree
      • Will show file difference. Changes tab will open on sidebar
    • or the current branch
      • An editor tab will open to show commit difference
  • see 38.2.16.6

38.2.14. Edits inside Editor

38.2.14.1. Line Comment C-/, Block Comment C-S-/
38.2.14.2. To Spaces / To Tabs (convert to Tab/Spaces for indentation) :: Find Action > To Tab
38.2.14.3. Surround with M-S-z or C-M-t or ⌥ ⌘ t
  1. Surround with Live Template
    • Select a text abc and wrap it into a function call
      • Settings > Live Templates > under user, +, Live Template >
        Abbreviation
        lifc
        Description
        Surround with function call
        Template text
        $END$($SELECTION$)
        Context
        Everywhere
    • Select the text, Code > Surround with, or M-S-z
  2. Surround with Emmet phpstorm:emmet:wrap
38.2.14.4. Code > Insert Live Template C-M-S-j
  • See 38.2.16.1.4
  • See 38.2.14.3.1
  • HTML
  • PHP
    pubf
    public function
    pubsf
    public static function
    prif
    private function
    prisf
    private static function
    prof
    protected function
    prosf
    protected static function
    thr
    throw new
    inco
    include once
    forek
    foreach(iterable_expr as $key => $value)
    eco
    echo
38.2.14.5. Postfix Template 38.2.16.1.4
38.2.14.6. Code > Generate… M-Insert
  • Generate PHPDoc Blocks
    • A manual way is go to the previous line and type /** Enter, PHPDoc will be generated
  • Constructor, Getter, Setter
  • Override methods
    • Don't have an option to override fields..
  • Implement methods
    • Magic methods
38.2.14.7. Code > Folding Fold Selection / Remove region C-.
  • Fold or unfold the parent of current code
  • Apply on integer in query to show the integer with thousand separator _ e.g. 1000 shows 1_000
38.2.14.8. Code > Move Line Down / Up ⇧ ⌥ Down
38.2.14.9. Code > Auto-Indent Lines (auto indent) :: ⌃ ⌥ i
38.2.14.10. Code > Analyze Code > Configure Current File Analysis :: ⌥ ⇧ ⌘ h

Turn off error highlighting, turn off highlight

38.2.14.11. Code > Code Completion
Basic
⌃ Space
  • Complete names of classes, methods and keywords within the visibility scope
  • Include Live templates
Type-matching
⌃ ⇧ Space
Complete Current Statement
⇧ ⌘ Enter
Cyclic Expand Word
⌥ / ⇧ ⌥ /
38.2.14.12. Show Context Actions (show intention actions) :: M-Enter ⌥ Enter
  • At code line, press M-enter
  • May provide some refactoring options
    • Convert to sprintf
    • Convert to string interpolation
    • if a Class function can be made to a static function
  • move cursor to arguments then hit the shortcut
  • Language Injection
    • For example, insert HTML code to a PHP variable. We need syntax highlighting of that HTML code inside the PHP file
    • https://www.jetbrains.com/help/phpstorm/using-language-injections.html#use-lang-annotation
    • Use PHP nowdoc or heredoc
      • Inside HereDoc Alt+Enter and choose Edit JavaScript Segment
    • Inline
      • Place cursor in front of a string, M-Enter > Inject Language using PHPDoc
    • $a = /** @lang JavaScript */ "console.log('hello')";
      Add a comment line above the variable assignment line
      // language=CSS
    • SQL, RegExp, XML, HTML, text
      • Place cursor in front of a string, M-Enter > Inject language or referrence to temporarily set the language
    • SQL Dialect
      • Settings > Editor > Language Injections

      • Manually find an entry named php: <<< SQL. Duplicate it and change SQL to MYSQL, now heredoc will use the MySQL dialect 

        $s=<<<MYSQL
SELECT * FROM `table`;
MYSQL;
38.2.14.13. Fix Doc Comment Find Action > fix doc comment
  • This is the only way to insert or fix JSDoc
38.2.14.14. Emmet
  1. How it works

    snippets.json store something like "m": "margin:|;"

    Settings > Emmet > HTML, CSS, JSX Settings > Keymap > search emmet

    https://docs.emmet.io/cheat-sheet/ https://docs.emmet.io/

  2. Surround with, wrap each line with a tag phpstorm:emmet:wrap

    Refer to phpstorm:surround with 

    <!--
    * Unordered item 1
    * Unordered item 2
    * Unordered item 3

    Select 3 lines and type (abbreviation)
    ul>li*
  -->

<ul>
  <li>* Unordered item 1</li>
  <li>* Unordered item 2</li>
  <li>* Unordered item 3</li>
</ul>

<!-- 
     Use filter t (trim) to remove list markers

     - ul>li*|t
     - ul.nav>li.nav-item$*>a|t
  -->
  3. Number $
  4. Repeat input * as $#
  5. Filters using Pipe(s)
    1. Escape XML e
      iframe|e
&lt;iframe src="" frameborder="0" title=""&gt;&lt;/iframe&gt;
    2. Comment c

      Add comments around tags with id or class attribute.

    3. xsl
    4. Single Line s
    5. Trim t
  6. HTML
  7. CSS
    1. margin (m), color (c), border (bd)
      • Length
        • m10 TAB for margin: 10px;
        • m10-20 TAB for margin: 10px 20px;
        • m-10–20 TAB for margin: -10px -20px;
        • m1.5 TAB for margin: 1.5em;
        • m1.5rem TAB for margin: 1.5rem;
        • m10p30e5x TAB for margin: 10% 30em 5ex;
        • percentage
        • em
        • ex
      • c#fc0
      • e.g. bd5#0s border: 5px #000 solid
      • fz
38.2.14.15. Start New Line Before Current ⌥ ⌘ Enter
38.2.14.16. Edit > Join lines C-S-j ⌃ ⇧ j
38.2.14.17. Edit > Toggle Case ⇧ ⌘ u
38.2.14.18. Window > Editor Tabs
  • Maximize Editor / Normalize Splits
38.2.14.19. Toggle Rendered View - rendered PHPDoc ⇧ ⌥ q

38.2.15. Settings > Directories

38.2.15.1. Exclude Directories
38.2.15.2. Resource Root
This should be set to where the final CSS files are placed
e.g. assets/css/style.css has background-image: url(../abc.jpg)
  • Then the Resource Root should be set to assets/css

38.2.16. Settings > Editor

38.2.16.1. General
  1. General
    • On Save
      • Turn off Remove trailing spaces
    • Soft Wraps
      Only show soft-wrap indicators for the current line
      default on, turn it off
  2. Code Completion
    • Parameter Info
      Show full method signatures
      default off
      • See 1
    • database auto completion
      Qualify object with
      database auto completion
  3. Code Folding
    • Fold by default
      • XML
        XML entities
        disable to always expand HTML entities e.g. show `&nbsp;` instead of a visual space
  4. Postfix Completion
    1. PHP
      array_walk($EXPR$, function ($v, $k) use (&$r) {
    $END$
}, $r = '');

      array_reduce($EXPR$, function ($carry, $item) use (&$externalParam) {
    $carry[] = $item;
    return $carry; 
}, array());

      array_filter($EXPR$, function ($v, $k) use (&$externalParam) {
    return $v > 0;
}, ARRAY_FILTER_USE_BOTH);

      // arrow
(function () {
    $EXPR$
})()

      array_map(function ($v) use (&$externalParam) {
    return $v;
}, $EXPR$);

      $END$($EXPR$ ?? null)
  5. Smart Keys
    Reformat on paste
    indent each line (default)
    Use "CamelHumps" words
    selection expansion and move caret to word using CamelHumps or underscore. Word separator / word boundary is underscore.
    (no term)
    PHP
    Enable smart function parameters completion
    off
38.2.16.2. Color Scheme
  • General
    • Code
      • Identifier under caret
      • Identifier under caret (write)
  • Language Defaults
    Semantic highlighting
    enabled
38.2.16.3. Code Style
  1. For each language
    • Use Project scheme
      • e.g. Set from WordPress Change Tab size and Indent from 4 to 2
    • Set a default Code Style Scheme e.g. Project sheme and Reformat code will use that scheme
      • For IDE scheme, stored at C:\Users\Li\AppData\Roaming\JetBrains\PhpStorm2020.1\codestyles or can export scheme
  2. PHP
    PSR-1 < PSR-2 < PSR-12
    Basic Coding standards https://www.php-fig.org/psr/psr-12/
    PSR-0 < PSR-4
    Namespaces, class names and file paths
    (no term)
    Use PSR-12 and do some manual changes
    • Code Generation
      Comment Code
      disable Line comment at first column, and enable Add a space at comment start
    • Wrapping and Braces
      • 'if()' statement
        • 'else' on new line
        • Force braces
      • try statement
        • catch on new line
        • finally on new line
  3. JavaScript
    • Punctuation
      • Use semicolon to terminate statements always
  4. HTML
    https://google.github.io/styleguide/htmlcssguide.html
    don't have an xml for phpStorm.. need to do it manually
    (no term)
    Other
    Do not indent children of
    add script
38.2.16.4. Inspections
  1. PHP Mess Detector (PHPMD)
    • PHP > Quality Tools > Mess Detector
      • Static code analyzer
      • If a remote interpreter is used, the default System PHP should not be used. Add one
        • The PHP Mess Detector path should be the path in the Docker container (remote)
        • Enable all rules under Options

      • To get the executable, you will create a new folder with one composer.json file 

        {
    "require-dev": {
        "phpmd/phpmd" : "@stable"
    }
}

        cd path/to/folder && composer install. The executable is path/to/folder/vendor/bin/phpmd

        • https://phpmd.org/download/
        • path/to/folder/vendor/phpmd/phpmd/src/main/resources/rulesets
          • cleancode.xml
          • codesize.xml
          • controversial.xml
          • design.xml
          • naming.xml
          • unusedcode.xml
          • my-custom-ruleset.xml
        • path/to/folder/vendor/bin/phpmd path/to/my.php my-custom-ruleset
    • Editor > Inspections > PHP > Quality tools > PHP Mess Detector validation
    1. Customize ruleset
      <?xml version="1.0"?>
<ruleset name="My first PHPMD rule set"
         xmlns="http://pmd.sf.net/ruleset/1.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://pmd.sf.net/ruleset/1.0.0
                     http://pmd.sf.net/ruleset_xml_schema.xsd"
         xsi:noNamespaceSchemaLocation="
                     http://pmd.sf.net/ruleset_xml_schema.xsd">
    <description>
        My custom rule set that checks my code...
    </description>

    <!-- Import the entire unused code rule set -->
    <rule ref="rulesets/unusedcode.xml" />

    <rule ref="rulesets/codesize.xml">
        <exclude name="CyclomaticComplexity" />
    </rule>
    <!--
        Import the entire cyclomatic complexity rule and
        customize the rule configuration.
    -->
    <rule ref="rulesets/codesize.xml/CyclomaticComplexity">
        <priority>1</priority>
        <properties>
            <property name="reportLevel" value="1" />
        </properties>
    </rule>

    <rule ref="rulesets/naming.xml" />
    <rule ref="rulesets/cleancode.xml" />
    <rule ref="rulesets/controversial.xml" />
    <rule ref="rulesets/design.xml" />

</ruleset>
  2. General
    • Skip duplicates simpler than:
      • Smaller the units, more duplicates will be found
  3. PHP
    • Refactoring opportunities
      • Complex class should be refactored
      • Complex function should be refactored
        Cyclomatic complexity
        default 15
      • Function has too high cyclomatic complexity
        Cyclomatic complexity
        default 20
      • LCOM metric
        • Lack of Cohesion of Methods
  4. SQL
    • Unresolved reference
      • Enable Suppress for quoted identifiers which may be treated as strings
        • But double quotes in INSERT ... VALUES still throw errors
  5. JavaScript and TypeScript
    • Function metrics
      • Overly complex function
        Cyclomatic complexity
        default 10
38.2.16.5. File and Code Templates
38.2.16.6. File Types
  • Ignore Files and Folders
    • Remove .git to show .git folder under Project View
      Also consider exclude this folder from index
      In Project View, select the .git folder and Search Action Excluded. Then Settings > Directories > shows the .git folder is excluded
38.2.16.7. Inlay Hints
Show name for all arguments
default off
(no term)
Always show pass-by-reference
(no term)
Find Action > Enable Parameter Name Hints or Disable Hints will turn on/off both of the above options
38.2.16.8. Duplicates
  • PHP
    • Detect duplicates with different: (ignore names and values)
      Variable or identifier names
      enabled
      Function or field names
      disabled
      Constant values
      disabled
      (no term)
      See 1
38.2.16.9. TODO
  • Default \b(todo|fixme)\b.*, show comments with those as prefix in window TODO M-6

38.2.17. Settings > Tools > Database

38.2.17.1. Data Editor and Viewer
  • Data Presentation
    Automatically transpose tables
    Always
  • Data Sorting
    Sort tables by numeric primary key
    Descending

38.2.18. Settings > Advanced Settings

  • Editor Tabs
    • Equalize proportions in nested splits

38.2.19. .editorconfig

root = true

[*]
end_of_line = lf

[*.js]
indent_size = 2

38.2.20. jsconfig.json

38.2.21. Refactor

Refactor This..
⌃ t
Rename
⇧ F6
Change Signature (change method/parameter name, parameter order, return type)
⌘ F6
Inline… (inline variable)
⌥ ⌘ n
  • Remove the variable assignment line and move the value into the place where the variable is called
  • Opposite of Extract Variable
(no term)
Make Static
(no term)
Extract/introduce
Variable
⌥ ⌘ v
Constant
⌥ ⌘ c
Method
⌥ ⌘ m
Field (extract to a class property)
⌥ ⌘ f
Parameter (extract to a function parameter)
⌥ ⌘ p
(no term)
Interface
  • Place caret at a class name, a new file of interface is created, and the current file will implement the newly created interface
(no term)
Extract Class
  • Place caret at a class method, select additional methods, extract them into another class
Pull members up / Push members down
move methods to parent/child class
38.2.21.1. Move Class F6

38.2.22. Help

38.2.22.1. Find Action ⇧ + ⌘ + a
  • On Mac, there's a shortcut key binding conflict. Turn it off or use Search Everywhere on PHPStorm instead S+S
38.2.22.3. Delete Leftover IDE Directories

38.2.23. Database

38.2.23.1. Setup for a project
  • Settings > Languages & Frameworks > SQL Dialects
    • Project SQL Dialect
      • Add specific files to use a different dialect from the project SQL dialect
  • See 2
38.2.23.2. Go to DLL :: ⌘ b
38.2.23.3. Select in Database explorer :: ⌥ ⇧ b
38.2.23.4. Full text search on a table :: ⌥ ⇧ ⌘ f
38.2.23.5. Data Editor (table view)
Edit Filter Criteria (go to where filter)
⌥ ⇧ ⌘ f
Columns List (popup)
⌘ F12
Go To Row…
⌘ l
Go to row #1
1
Go to column abc at row #1
abc:1
38.2.23.6. Console

When there're multiple statements (multiple ;), C-Enter (or to Execute from Menu) prompts a list of statements to run:

  • Smallest statement
    • The smallest of the possible statements is executed. For example, when the cursor is inside a subquery, the subquery is executed.
  • Largest statement
    • The largest possible statement is executed.
    • e.g., when the cursor is inside a subquery, an outer statement is executed
  • Largest statement or batch
    • For Transact-SQL (SQL Server and Sybase), the current batch of statements is executed. For all other dialects - the same as the previous option.
  • Whole script. All the statements are executed. (This item always appears and always appear last)
  • You can also C-a to select all statements and C-Enter to run all of them
38.2.23.7. Return all query results
  • Click on the Wrench icon on the toolbar of the Database Console tool window
  • Switch to Database | Data Views page, specify 0 in the Result set page size field, and click OK
  • Default is 500
  • That way you can export all rows to csv file
38.2.23.8. Icons
  • https://www.jetbrains.com/help/phpstorm/database-tool-window.html
  • Default description appears next to each column shows
    • bigint(20) unsigned = 0
    • bigint(20) unsigned (auto increment)
    • But it doesn't show whether it is NOT NULL, indexed or primary/foreign key. Need to rely on icons
    • a circle on the bottom left
    • the left bar is highlighted (in blue)
    • yellow key
    • blue key
38.2.23.9. sql_mode or other session variables are not followed by global variables
Reason
JDBC driver might override and set session variables differently than global variables set on MySQL target server
Solutions
choose 1
Change on the Data Source (connection)
worked for me 20220831
  • Options > Startup script
    • SET SESSION sql_mode='NO_ENGINE_SUBSTITUTION'
(no term)
Change on the Drivers (NySQL)
  • Advanced > sessionVariables
    • sql_mode='NO_ENGINE_SUBSTITUTION'

38.2.24. Plugins

38.2.24.1. Makefile support
38.2.24.2. Makefile Navigator

Turn off the plugin:Makefile support and enable this plugin

38.2.24.3. Themes
  1. One Dark theme
    • There're 2 separate theme settings in PhpStorm:
      • Main theme used in UI and Editor: Settings > Appearance & Behavior > Appearance > Theme
      • Theme specifically for Editor: Settings > Editor > Color Scheme > Scheme
      • If there's any problem, make sure these 2 settings use the same theme
  2. Cyan Light Theme
  3. Atom One Theme
  4. Dark Purple Theme
  5. Gradianto
  6. Hiberbee Theme
38.2.24.4. Live Edit
38.2.24.5. JSON Sorter
38.2.24.6. PostCSS
38.2.24.7. Unit File Support (systemd)
38.2.24.8. .ignore
38.2.24.9. Ideolog

Interactive viewer for '.log' files

38.2.24.10. Uninstalled
  1. Rainbow Brackets
  2. Atom OneDark Theme (updated 3 years ago)
  3. Xcode-Dark Theme
  4. CodeTogether
    • Remote pair programming
  5. TextMate

    Install plugin TextMate bundles support Settings > Editor > TextMate Bundles Add a git repo to support Python https://github.com/textmate/python.tmbundle

  6. JS GraphQL
  7. Python
  8. nodejs phpstorm:nodejs

    Enable ES6 for Settings > Languages & Frameworks > Javascript > JavaScript language version > ECMAScript 6 Install plugin NodeJS, restart phpStorm after Node and NPM are installed on Windows. Enable Node.js Core library in Settings > Languages & Frameworks > Node.js and NPM Node interpreter looks like C:\Program Files\nodejs\node.exe

  9. Golang phpstorm:golang

    Search for Go, install plugin and restart phpStorm Setup SDK, which is c:\go

  10. Org4Idea
38.2.24.11. Presentation Assistant

38.2.25. Docker support

In Docker for Windows > Settings > General Choose Expose daemon on tcp://localhost:2375 without TLS

In phpStorm, Settings > Build, Execution, Deployment > Docker, create Docker with a + button.

Default should work, so just OK

API URL: tcp://localhost:2375 Certificates folder: blank Docker Compose executable: docker-compose VirtualBox shared folders: VM Path - /c/Users, Local path - C:\Users

You should see Docker under the bottom bar, select and connect to it. Now you should see the running containers and available images.

38.2.26. Vue.js

JetBrains plugin repository

38.2.27. WordPress Support

Settings > Languages & Frameworks > PHP > Frameworks > WordPress > Enable WordPress integration

  • Specify installation path
  • May also need to add the installation path to Settings > Languages & Frameworks > PHP > Include Path

38.2.28. Xdebug

38.2.28.1. Basics
  • Since XDebug 2.0, protocol DBGp is used
  • Set up xdebug:config
  • Restart the image:php:7.x-fpm container or apache:restart or nginx:restart
  • When one of these happens, XDebug server will try to connect IDE debugger:
    • URL
      • XDEBUG_SESSION_START=session_name
      • XDEBUG_PROFILE=1
    • XDEBUG_SESSION_START as POST parameter
    • a cookie named XDEBUG_SESSION
    • export XDEBUG_CONFIG="idekey=session_name remote_host=localhost profiler_enable=1" in XDebug host
  • Start Listening for PHP Debug Connections on phpStorm
    • IDE's debugger works by listening the remote_port. It works without setting up any CLI interpreter
    • Only enable Use path mappings and specify the Path Mapping. Leave defaults for Name:_, Host:_, Port:80, Debugger:Xdebug
    • The added server can be found later under Language & Frameworks > PHP > Servers
38.2.28.2. For lando:xdebug

For per project to include third-party code (e.g. libraries) that is used for completion and reference resolution in functions/methods that use file paths as arguments, for example, require() or include()

Settings > Languages & Frameworks > PHP

  • choose the correct php version
  • Add CLI interpretor > remote is Docker, for Server, if Docker is not available, hit New > Unix socket
    Image name
    choose the image with the corresponding PHP version e.g. devwithland/php:7.0-fpm for Pantheon project
    The following fields are by default
    PHP executable: php. Hit Apply
  • and then include:
    • ~/.lando/services/config/pantheon or Windows C:\Users\li\.lando\services\config\pantheon
    • Or other recipe C:\Users\li\.lando\services\config\drupal8
    • The point is to load prepend.php which defines global vars related to environments
  • Start Listening for PHP Debug Connections, then load the local website, accept the Incoming Connection from XDebug
  • Settings > Languages & Frameworks > PHP > Servers will show the newly accepted connection as a server. Make sure the following is correct
    • host: appserver, port: 80, debugger: XDebug
    • project files: your-local-directory-of-the-project, absolute path on server: /app (very important)
    • include path: C:\Users\li\.lando\services\config\pantheon, absolute path on server: /srv/includes
38.2.28.3. For Vagrant
  • Settings > Languages & Frameworks > PHP
    • Choose correct PHP version
    • Install PHP on Vagrant 1.1.1
    • Add CLI Interpreter > type is Vagrant, choose the path of the Vagrantfile. Everything should be auto set up:
      • Vagrant Instance Folder: C:\c
      • Varant Host URL: ssh://vagrant@127.0.0.1:2222
      • /usr/bin/php7.3
      • Configuration file: /etc/php/7.3/cli/php.ini
  • Settings > Languages & Frameworks > PHP > Quality Tools
    • Refer to #php:composer:php_codesniffer
    • Then Editor > Inspections > PHP > Quality Tools > enable PHP_CodeSniffer validation, and choose one coding standard
  • Run Code > Inspect Code > select scope, double check Quality Tools settings, run
    • PHP CODE BEAUTIFIER AND FIXER auto fix is available to some problems listed under PHP > Quality Tools > PHP_CodeSniffer validation
38.2.28.4. For Docker > Vagrant > Windows
  • Do the same as vscode:remote:ssh:proxy
  • Settings > Language & Frameworks > PHP
    • Choose correct PHP version
    • New CLI Interpreter, type SSH Credentials
      • Host, Port, User name have to be filled. Same as Windows local ~/.ssh/config for remotecontainer
      • Auth type: OpenSSH config and authentication agent
      • Select the CLI php. Find it in Docker container by which php
    • <Project root> -> /var/www/html
38.2.28.5. For Docker
  • PHP
    PHP Language Level
    7.3
    (no term)
    CLI Interpreter
    Type
    Docker Compose
    Server
    Docker for Mac
    Configuration files
    the docker-compose.yml file
    Service
    the service/container name. Auto prefilled as docker compose yml is loaded.
    Lifecycle
    Connect to existing container
    PHP executable
    will be auto populated
    Path Mappings
    will be auto populated
38.2.28.6. XDebug config 
# Type configname = default
# boolean xdebug.collect_vars = false 
# integer xdebug.show_local_vars = 0

# boolean xdebug.remote_enable = false
xdebug.remote_enable = 1

# If it's 1, remote_host will be ignored. Xdebug on server will try to connect debugger e.g. IDE using $_SERVER['REMOTE_ADDR'] and xdebug.remote_port 
# This remote_connect_back only works for HTTP but not CLI request
# xdebug.remote_connect_back = 1

# Docker container hosted on Vagrant and IDE is on Windows. Docker container with Xdebug server can connect to Windows using an IP
# If Xdebug is on a Docker container hosted directly on Windows, consider use `host.docker.internal` or `localhost`
xdebug.remote_host=10.0.2.2

# default. Xdebug server will try to combine remote_host or $_SERVER['REMOTE_ADDR'] and this remote_port to connect the debugger e.g. on IDE over the internet
# No port forwarding should be set up as the connection is over the internet
# xdebug.remote_port=9000

# req :: default. initiates a session as soon as script is started
# jit :: initiate when a session should only be initiated on an error 
# xdebug.remote_mode = req

# Default. Xdebug server does not attempt to try to connect to a debug client e.g. IDE
# xdebug.remote_autostart=false

# Shows whether or not Xdebug server can connect to a debug client
# xdebug.remote_log=/tmp/xdebug.log

# xdebug.profiler_enable_trigger = 1

38.2.29. PHPUnit

  • php:phpunit
  • Set up CLI interpreter as Local or phpstorm:php:cli:ssh proxy
  • Languages & Frameworks > PHP > Test Frameworks, add a PHPUnit Local or

    • PHPUnit by Remote Interpreter

      • Select the previously set remote CLI Interpreter, then Path Mappings is auto set
      • PHPUnit library: Use Composer autloader
      • Path to script: /var/www/html/vendor/autoload.php
        • Test Runner
      • Default configuration file: /var/www/html/phpunit.xml
  • Run > Edit Configuration, add PHPUnit
    • Give a name and only check Defined in the configuration file
  • Now click the run button!

38.2.30. Troubleshoot

38.2.30.1. TS: Reset Index

File | Invalidate Caches and restart IDE

38.2.30.2. TS: Page '…js.map' requested without authorization

Settings -> Build, execution, development -> Debugger - >Allow unsigned request

38.2.30.3. SQLite cannot open local file
  • Run as admin to open phpStorm

38.3. Emacs

38.3.2. Install & Configuration

38.3.2.1. Windows
  • Download for Windows https://ftp.gnu.org/pub/gnu/emacs/windows/
  • https://ftp.gnu.org/gnu/emacs/windows/emacs-26/emacs-26.1-x86_64.zip
    Previous versions
    emacs-26.1-x86_64.zip (with all deps 64-bit) emacs-25.1-i686-w64-mingw32 emacs-24.5-bin-i686-mingw32.zip
  • Extract it to C:\emacs, so you will have C:\emacs\emacs-24.5-bin-i686-mingw32\
  • Windows Shortcut to Start
    • Make a shortcut of this file /bin/runemacs.exe and copy to your notes folder say C:\c\notes\
    • Then right click, select Properties of the shortcut and change the Start In to C:\c\notes
  • Windows Batch File to Start
    • Set a HOME user environment variable in Windows as C:\c\notes and that is ~/ as the home directory inside Emacs
    • Include an init file C:\c\notes\.emacs

    • Home folder in Emacs is C:\c\notes and from time to time there're auto-save-list (auto saved files records) saved in home folder C:\c\notes\.emacs.d folder #+NAME C:/c/notes/run.bat 

      set HOME=E:\li\notes
C:\emacs\emacs-25.1-i686-w64-mingw32\bin\runemacs.exe notes.org
  • Extract emacs-25-i686-deps.zip and copy all files in bin to C:\emacs\emacs-25.1-i686-w64-mingw32\bin
38.3.2.2. Install on Ubuntu 18.04
sudo apt update
sudo add-apt-repository ppa:kelleyk/emacs
sudo apt install emacs26 -y
emacs notes.org

Install linux:font, open Emacs:

  • Options > Set Default Font to choose a font
  • Options > Save Options
  • M-x customize-face RET default RET, click save and apply
  1. Install for terminal only
    • sudo apt update && apt upgrade -y and reboot
    • sudo apt install build-essential libncurses-dev
    • Go to Emacs Download page to get the link. e.g. wget http://mirrors.syringanetworks.net/gnu/emacs/emacs-26.1.tar.gz
    • tar -xzvf emacs-26.1.tar.gz
    • cd emacs-26.1/ ./configure --without-x --with-gnutls=no
    • make and sudo make install
    • Run emacs emacs
    • Change config emacs ~/.emacs
    • Copy https://melpa.org/#/getting-started and change the line from (proto (if no-ssl "http" "https"))) to (proto (if no-ssl "http" "http")))
    (require 'package)
(let* ((no-ssl (and (memq system-type '(windows-nt ms-dos))
            (not (gnutls-available-p))))
   (proto (if no-ssl "http" "http")))
  (when no-ssl
    (warn "\
Your version of Emacs does not support SSL connections,
which is unsafe because it allows man-in-the-middle attacks.
There are two things you can do about this warning:
1. Install an Emacs version that does support SSL and be safe.
2. Remove this warning from your init file so you won't see it again."))
  ;; Comment/uncomment these two lines to enable/disable MELPA and MELPA Stable as desired
  (add-to-list 'package-archives (cons "melpa" (concat proto "://melpa.org/packages/")) t)
  ;;(add-to-list 'package-archives (cons "melpa-stable" (concat proto "://stable.melpa.org/packages/")) t)
  (when (< emacs-major-version 24)
    ;; For important compatibility libraries like cl-lib
    (add-to-list 'package-archives (cons "gnu" (concat proto "://elpa.gnu.org/packages/")))))
(package-initialize)
    • emacs notes.org and
      • [M-x] package-refresh [RET]
      • [M-x] package-list-packages [RET]
38.3.2.3. Install on Mac Catalina
  • brew install --cask emacs
  • macos:full disk access to Emacs">Applications > Emacs
  • Troubleshoot
    Emacs 29.4 on Mac

    Warning: arch-dependent data dir 'Contents/MacOS/libexec/': No such file or directory 

    # shows the installation path
which emacs
cd /Applications/Emacs.app/Contents/MacOS
cp -a libexec-arm64-11 libexec
38.3.2.4. Init file
  • ~/
    .emacs or .emacs.el
    old location for config file
    .emacs.d/init.el
    macOS and Windows
    .config/emacs/init.el
    for Linux

  • Use run.bat to change the HOME directory 

    #!/bin/bash
HOME=~/li/notes emacs -L . --debug-init notes.org
  • M-: (load user-init-file) RET
38.3.2.5. Get version :: M-x info M-x org-info
  1. Releases
    28.1
    https://www.masteringemacs.org/article/whats-new-in-emacs-28-1
38.3.2.6. Command line options

38.3.3. Theme

M-x customize-themes
Load a theme
(no term)
M-x load-theme RET modus-vivendi RET

38.3.4. Keybinding

M-
Alt, ⌥
C-
Ctrl, ⌃
S-
Shift, ⇧
s-
Special ⌘

38.3.5. MELPA packages

  • Add http://melpa.org/packages/
  • Run M-x list-packages to see all packages including built-in pkgs and package-archives
    i
    installation
    u
    unmark
    x
    perform the i and u installation
    U
    mark available upgrades. U x is to upgrade all upgradable packages
    (no term)
    RET to see more about a pkg
    q
    quit
    C-x k
    kill current buffer
    M-x package-refresh-contents
    to refresh the package list
  • New install is in ~/.emacs.d/elpa/newpkgname-YYYYMMDD.123
    • May need to follow instructions on Melpa.org to initialize in .emacs init file
  • Sometimes the compilation of a package dependes on the core Emacs. If the core is updated, a package may need to be reinstalled
    • M-x package-reinstall > php-mode

  • In .emacs, run package-install-selected-packages. The following message 1 Packages are not available (the rest already installed). To find out which packages are not available, paste this in *scratch* buffer and run it C-j: 

    (let* ((not-installed (seq-remove #'package-installed-p package-selected-packages))
      (unavailable (seq-filter (lambda (p) (not (assq p package-archive-contents)))
                               not-installed)))
   unavailable)

38.3.7. GitHub pkg el-get

It can be used to install/load pkgs from difference sources :: https://mrblog.nl/emacs/config.html

Place packages under load-path C-h v load-path RET C:\emacs\emacs-24.5-bin-i686-mingw32\share\emacs\VERSION\site-lisp is /usr/local/share/emacs/VERSION/site-lisp for a particular Emacs version C:\emacs\emacs-24.5-bin-i686-mingw32\share\emacs\site-lisp is /usr/local/share/emacs/VERSION/site-lisp for all Emacs version

*.el (source) *.,elc (byte-compiled) are ELPA package and should be used for emacs 24+ 

(add-to-list 'load-path "~/.emacs.d/lisp/")

(add-to-list 'load-path "c:/c/notes/lisp")

38.3.8. M-x: run a command

customize-variable
see a variable value and may change later M-x customize-variable [RET] package-archives [RET]
q
exit the buffer
emacs:m-x:delete-trailing-whitespace
delete current buffer's trailing whitespace
emacs:m-x:indent-region
reformat indentation for the current buffer without select regions. Or select region then C-M-\
Show init time
emacs:m-x:emacs-init-time
(no term)
Use Customize UI to save variables in .emacs emacs:m-x:customize-option
  • Variable will be set under (custom-set-variables in .emacs
visual-line-mode
toggle
replace tab with whitespaces
M-x tabify M-x untabify
Toggle minor mode
M-x debug-mode

38.3.9. Help :: C-h

minor / major modes being used
C-h m
Exit Help buffer
C-x 1
See all help options
C-h C-h or C-h ?
Find shortcut by command where-is
C-h w
Find command by shortcut describe-key
C-h k
Description of a function
C-h f
Describe symbol (variable and function)
C-h o
Get variable value
C-h v
Normal hooks
hook names end with -hook
Abnormal hooks
end in -functions

38.3.10. Basics

   
Editing  
underlined  
verbatim  
verbatim  
strike_through  
Italics  
   
C-c . Insert timestamp
   
M-Backspace Kill the previous word
   
C-S-Bksp Kill whole line
   
<q BEGIN_QUOTE
<H BEGIN_HTML
<c BEGIN_CENTER
<e BEGIN_EXAMPLE
c-c ; BEGIN_COMMENT
   
Headline  
TAB Local tree fold/unfold
C-u TAB Global tree fold/unfold
C-u C-u C-u TAB Show all
   
Table emacs:table
C-c - Insert a horizontal line below
M-S-<down> Insert a new row above
C-c <RET> Insert horizontal line and a new row below
M-S-<up> Delete a current row
   
M-S-<right> Insert new column on the right
M-S-<left> Delete current column
   
C-c <SPC> Blank current field
C-c ` Edit current field
   
S-<TAB> Move to previous field
M-<up>/<down> Move a row
M-<left>/<right> Move a column
   
C-c | After text is selected, convert to table
   

38.3.11. Temporarily Change Font Size in Current Buffer

Increase
C-x C-+
Decrease
C-x C--
Restore the default (global) font size
C-x C-0

38.3.12. Windows

Remove current window
C-x 0
Remove a help window (delete-other-windows)
C-x 1
Split horizontally
C-x 2
Split vertically
C-x 3
Move to other window
C-x o
(no term)
Scroll page down (scroll-other-window) and up in other window (scroll-other-window-down):: C-M-v, S-C-M-v
Switch to the other buffer switch-to-buffer
C-x b

38.3.13. Search and replace

C-s
search by pure text, next search result. If search string contains no uppercase, then it's a case-insensitive search
C-r
previous search
M-C-s
regex search
manage_.+_columns
search manage_*_columns
M-C-r
regex reverse search
M-s o
list-matching-lines regex and show matching lines
C-g
abort search and go to the original position
C-s C-s or C-r C-r
search again
Regex escape
[ ] \ + * ? .
Regex case-sensitive
case-sensitive is auto enabled when text contains uppercase
(no term)
Emacs Regex
M-x replace-string
Search and replace from the current cursor to the end of buffer

38.3.14. Region, Selection, Copy & Paste

C-space
start region, then move cursor
C-x h
copy all in buffer, select all
M-@
mark-word. select next word. Repeat to expand
M-h
mark-paragraph. Repeat to expand
C-M-h
mark-defun. select a function
C-g
keyboard-quit quit selection
C-w
cut
C-k
cut from the cursor to the end of the line (not including the newline character)
M-w
copy without cut (copy to kill-ring, shared with other buffers)
C-y
yank (paste)
C-Backspace, M-Backspace
kill one word backwards at the cursor and to kill-ring
M-d
kill one word forwards at the cursor and to kill-ring
C-S-Backspace or C-S-Delete
kill current line and add to kill-ring
C-a
move to the beginning of line
C-n
move to next line
C-space
start ring
C-a C-k C-/ or C-a C-Space C-e M-w
copy current line without the newline and buffer remains unchanged
C-a C-Space C-n M-w
copy current line including the newline and buffer remains unchanged
(no term)
Duplicate the current line
Method 1
C-S-Delete > C-y > C-y
(no term)
Method 2
  • C-= to expand region to select the whole line (keep expanding then several lines can be duplicated)
  • cut whatever is marked
  • C-y
Method 3
C-a > C-k > C-k > C-y > C-y
Easy to remember
Control + akky
C-/, C-_ or C-x u
undo
To redo after an undo
C-g C-_

38.3.15. Moving and basic editing

C-l
Scroll current line to center
C-x b
switch to another buffer
M-<
Move to beginning of buffer e.g. top of file
M->
Move to end of buffer
C-a
go to beginning of line
C-e
go to end of line
C-n
Go to next line
C-p
Move to previous line
C-f
forward like <right>
C-b
backward like <left>
C-v
Page down
M-v
Page up
M-b
Move backward one word
M-f
Move forward one word
M-g M-g
Go to a line
C-c C-j
org-goto. Search in headlines only
(no term)
Go back / go to previous location in the local file
  • C-u C-Space
    • Jump to the previous mark and remove it from the mark ring
  • same as the above in case C-Space on Windows trigger Input Source change
M-^
join the current line with previous line

38.3.16. Headline

C-u TAB
Global tree fold/unfold
  • While Tab can show the first level child headings, this will show all the nested sub headings
  • Show all
S-<TAB>
In table, go to previous cell. In other places, global fold/unfold
C-c C-n
Next heading
C-c C-p
Previous heading Go to parent heading if triggered in the content of the heading
C-c C-f
Next heading same level
C-c C-b
Previous heading same level
C-c C-u
Backward to higher level heading outline-up-heading
M-<RET> or C-<RET>
insert heading
C-<RET>
org-insert-heading-respect-content
M-<right>/<left>
Decrease/increase level of a parent item only
M-S-<right>
Increase heading level of an item and its child items
M-S-<left>
Same as above but decrease
C-c *
turn a list into heading/headline, C-c - to turn a heading into list
C-x n s
Work on one section
C-x n w
Back to outline
C-c C-j
jump to different heading. C-g to exit and return to the current position. up/down to move headline, / sparse-tree search
(no term)
M-x
org-fold-show-branches or -f-br
fold content and only show headings and subheadings
(no term)
lili-show-path
(no term)
M-:
  • (org-get-outline-path t)
38.3.16.1. Custom Variables
org-adapt-indentation
nil. Don't indent body, subheadings and bodies of subheadings
  1. org-startup-folded

    Use startup lines on top of the file to avoid setting this variable 

    #+STARTUP: fold              (or ‘overview’, this is equivalent)
#+STARTUP: nofold            (or ‘showall’, this is equivalent)
38.3.16.2. Property Drawer
Set a property and create the property drawer if not exists
C-c C-x p
Set CUSTOM_ID property for a headline
CUSTOM_ID
  • CUSTOM_ID is used for HTML element ID in HTML export
M-x org-id-get-create or M-x o-get
insert a property drawer with the property ID
  1. I tried the following to remove random HTML id's and add anchor for headlines using CUSTOM_ID, but I failed. It creates performance issue
    (require 'org-id)
;; (setq org-id-link-to-org-use-id 'create-if-interactive-and-no-custom-id)

;; WARNING! Overwrite core function.
(defun org-id-new (&optional prefix)
  "Create a new globally unique ID.

An ID consists of two parts separated by a colon:
- a prefix
- a unique part that will be created according to `org-id-method'.

PREFIX can specify the prefix, the default is given by the variable
`org-id-prefix'.  However, if PREFIX is the symbol `none', don't use any
prefix even if `org-id-prefix' specifies one.

So a typical ID could look like \"Org-4nd91V40HI\"."
  (let* ((prefix (if (eq prefix 'none)
         ""
           (concat (or prefix org-id-prefix) "-")))
     unique)
(if (equal prefix "-") (setq prefix ""))
(cond
 ((memq org-id-method '(uuidgen uuid))
  (setq unique (org-trim (shell-command-to-string org-id-uuid-program)))
  (unless (org-uuidgen-p unique)
    (setq unique (org-id-uuid))))
 ((eq org-id-method 'org)
  (let* ((etime (org-reverse-string (org-id-time-to-b36)))
     (postfix (if org-id-include-domain
              (progn
            (require 'message)
            (concat "@" (message-make-fqdn))))))
    (setq unique (concat etime postfix))))
 (t (error "Invalid `org-id-method'")))
(concat prefix unique)))

(defun eos/org-custom-id-get (&optional pom create prefix)
  "Get the CUSTOM_ID property of the entry at point-or-marker POM.
   If POM is nil, refer to the entry at point. If the entry does
   not have an CUSTOM_ID, the function returns nil. However, when
   CREATE is non nil, create a CUSTOM_ID if none is present
   already. PREFIX will be passed through to `org-id-new'. In any
   case, the CUSTOM_ID of the entry is returned."
  (interactive)
  (org-with-point-at pom
(let ((id (org-entry-get nil "CUSTOM_ID")))
  (cond
   ((and id (stringp id) (string-match "\\S-" id))
    id)
   (create
    (setq id (org-id-new (concat prefix "h")))
    (org-entry-put pom "CUSTOM_ID" id)
    (org-id-add-location id (buffer-file-name (buffer-base-buffer)))
    id)))))

(defun eos/org-add-ids-to-headlines-in-file ()
  "Add CUSTOM_ID properties to all headlines in the current
   file which do not already have one. Only adds ids if the
   `auto-id' option is set to `t' in the file somewhere. ie,
   #+OPTIONS: auto-id:t"
  (interactive)
  (save-excursion
(widen)
(goto-char (point-min))
(when (re-search-forward "^#\\+OPTIONS:.*auto-id:t" (point-max) t)
  (org-map-entries (lambda () (eos/org-custom-id-get (point) 'create))))))
38.3.16.3. Drawer
C-c C-x d
Create a drawer

This is inside the drawer

38.3.17. Code Block

Create a code block
<e and Tab
Since Org mode 9.2
use C-c C-,
Inside a code block
C-c + ' to go to a new major-mode edit buffer to edit the code, C-x C-s saves the buffer and updates the contents of the Org buffer. C-c + ' again to exit
(no term)

Inline code block 

src_<language>[<header arguments>]{<body>}

src_sh{echo "hello"}

src_html[:exports code]{<el></el>}

src_haskell[:exports both]{fac 5}
All languages
html xml css sass js makefile java sh sql sqlite ruby python perl C C++ matlab org dns
(no term)
Find modes on GitHub -mode to *-mode.el files
(no term)
When in *-mode edit buffer:

38.3.18. Export to HTML C-c C-e h h

  • Add markdown and Confluence export format
    • M-x customize-option > org-export-backends
      • Arrow down to md and Confluence and Enter
      • Arrow back to the top to Apply and Save then Enter, q to quit

  • Include that in the org file that you want to export. 

    #+SETUPFILE: my-setup.setup
    • https://orgmode.org/manual/Export-Settings.html
    • Include export keywords
      #+OPTIONS: num:nil
      overrides global variable org-export-with-section-numbers
      t
      default
      n (an integer)
      numbering will only happen for headlines whose relative level is higher or equal to n

38.3.19. Symbol and Escape emacs:escape

\vert{}
which escapes |

Table of Symbols

Insert a tab character Ctrl+q tab

38.3.20. Superscript and subscript

To toggle display: C-c C-x \
x_{sub script}
y^{super script}

38.3.21. TODO

  • Only applied to headlines not list items
  • Cycle through TODO, DONE and empty
  • M-S-return, M-x org-insert-todo-heading
    C-c C-c
    toggle checkbox state

38.3.22. Sparse Tree, Search

C-c / r
Search regex, C-c C-c to quit
M-g M-n
Next match
M-g M-p
Previous match

38.3.23. Link 

Link is defined in this order
[[#my-custom-id]]
- the entry with `CUSTOM_ID` property set to `my-custom-id`. Don't use because of performance issue

[[My Target]]
- By defalt, it leads to a text search. `My Target` is a headline with the exact text My Target
- Added to Mark Ring go back by using C-c &
- Won't work if the headline has dedicated target e.g. <<My Target>>
- Works even inside code block!

[[*Some section][Link Description]]
- same as [[My Target]] but you can use M-Tab to autocomplete headline
- does not work on Windows because keyboard shortcut conflict

[[internal target]] :: dedicated target
- <<internal target>>
  - Number in front of the internal target will be shown in the places that refer to this target
- NAME keyword that is put before the element it refers to [[Table One]]
  - CAPTION keyword is mandatory in order to get proper numbering

#+NAME: Table One
| a  | table      |
|----+------------|
| of | four cells |
|----+------------|

emacs:radio 

<<<Radio target>>>
This causes each occurence of 'Radio target' in normal text to become activated as a link
Emacs only update radio targets when the file is first loaded. To manual update, C-c C-c

38.3.24. SSH remote edit files emacs:ssh:tramp

  • Basics
    • C-x C-f and type /ssh:hostsetinsshconfig:/pathtofileordir
  • Windows
    • Need to use Putty (GUI to set a session) and Plink (CLI SSH client used by Emacs). Use this command /plinkx:sessionsetinputty:/pathtofileordir
    • Need to start up Emacs using PowerShell
    • Add the folder of plink.exe to env. var PATH. Check $env:PATH in PowerShell

38.3.25. Superscript and Subscript

By default Sometext_subscript and Sometext^superscript shows sub/super scripts when .org is exported.

Have this setting: Org > Customize > Browse Org Group > Group Org Export > Group Org Export General > Option Org Export With Sub Superscripts

After that, to show sub/super scripts, type these Sometext_{subscript} Sometext^notsuperscript^{superscript}

38.3.26. Include image

[[[[file:images/abc.jpg]]]]
And C-c C-x C-v to toggle display images on Emacs

38.3.27. dot

And add this to .emacs 

(org-babel-do-load-languages
'org-babel-load-languages
'((dot . t)))

To compile C-c C-c and display inline image C-c C-x C-v

dot_success.png

Graphviz Node, Edge and Graph Attributes

rankdir
default layout is from top to bottom. LR is left to right
splines
how edges look
none or ""
no edges
false or line
edges can go through nodes
true or spline
edges are routed around nodes (splines) and edges can be curved in order to avoid going through nodes
curved
edges are curved arcs
polylines
edges are one or multiple straight lines connected
ortho
edges are routed with 2 perpendicular straight lines

38.4. VS Code

38.4.1. Install, Basics

Ubuntu
sudo snap install --classic code
(no term)
Mac
  • brew install --cask visual-studio-code
  • C-S-p, select Shell Command: Install 'code' command in PATH
Updates and Blogs
https://code.visualstudio.com/updates
  • On Windows, I found I need to admin open to update VS Code

38.4.2. Extensions

  • ext install extension.name
  • You can disable one extension on User level and just enable it for the current workspace
38.4.2.1. GitLens eamodio.gitlens
  1. Git History donjayamanne.githistory
38.4.2.2. Live Server
38.4.2.3. Live Share
38.4.2.4. Remote SSH ms-vscode-remote.remote-ssh
  1. Basics
    • Basics
      • Install SSH client on local machine. Run command in VS Code Remote-SSH: Connect to Host...
      • After successful login, open Remote Explorer in left View Bar
        • Set up forward port
      • File > Close Remote Connection
    • Install Local Extensions on Remote
      • Remote Explorer > LOCAL - INSTALLED, the cloud download icon
    • Limitations
      • Alpine Linux and non-glibc based Linux SSH remote host is not supported
    • https://code.visualstudio.com/docs/remote/ssh
    • https://code.visualstudio.com/docs/remote/troubleshooting#_ssh-tips
      • Some files including extensions are installed in remote host ~/.vscode-server. Local VS Code is at ~/.vscode
      • VS Code nees execute access on /tmp directory on remote host
  2. Local > Vagrant > Docker Container
    Goal
    VS Code remote development on a Docker container, which is spawn by Vagrant (Docker host) using Remote - SSH VS Code extension
    (no term)
    Install linux:ssh:install in the target container
    (no term)
    On Docker host (Vagrant)
    (no term)
    On Windows
    • Copy the private key generated on Docker host to Windows ~/.ssh/id_rsa_vagrant_spawn
    • vagrant:ssh:pure ssh and test ssh myvagrant

    • ~/.ssh/config 

      # This is added by `vagrant ssh-config --host myvagrant >> ~/.ssh/config`
Host myvagrant
HostName 127.0.0.1
User vagrant
Port 2222
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
PasswordAuthentication no
IdentityFile C:/c/.vagrant/machines/default/virtualbox/private_key
IdentitiesOnly yes
LogLevel FATAL

Host remotecontainer
Hostname 172.17.0.2
User root
# the private key is copied from Docker host (Vagrant)
IdentityFile ~/.ssh/id_rsa_vagrant_spawn
# On Windows after `ssh -V` is v8+, just use `ssh` not the whole path of the executable
ProxyCommand C:\WINDOWS\System32\OpenSSH\ssh.exe myvagrant -W %h:%p

Host remotecontainer2
Hostname 172.17.0.2
User root
IdentityFile ~/.ssh/id_rsa_vagrant_spawn
# On Windows after `ssh -V` is v8+, use ProxyJump
ProxyJump myvagrant
    • done!
    (no term)
    TS: may need to linux:ssh:known_hosts:remove
38.4.2.5. Rest Client humao.rest-clident
  • Create .http file with GET http://a.ca HTTP/1.1
  • Generate target code to execute that REST request
38.4.2.6. Better Comments aaron-bond.better-comments
38.4.2.7. Prettier - Code formatter
38.4.2.8. Bracket Pair Colorizer 2
38.4.2.9. Python extension for Visual Studio Code vs:code:ext:my-python.python
Select code and run
S-Enter
(no term)
Linting
  • Default is PyLint
  • Change to PEP8
    • pip install autopep8
    • In VS Code, C-, search python formatting provider change it to autopep8
(no term)
Command
  • Select interpreter
  • Run Python File in Terminal
38.4.2.10. Python Docstring Generator njpwerner.autodocstring
38.4.2.11. JavaScript (ES6) code snippets xabikos.javascriptsnippets
38.4.2.12. IntelliJ IDEA Keybindings
name
k–kato.intellij-idea-keybindings
38.4.2.13. PHP related extensions
  1. PHP Tools for Visual Studio Code
  2. PHP Intelephense
    • Follow the Quick Start to disable some builtin extensions
    1. Other extensions depend on PHP Intelephense
  3. PHPDoc Comment
    • Shortcut
      Add PHPDoc Comment
      ⇧ ⌘ i
38.4.2.14. ESLint
See here for PhpStorm and ESLint
https://www.jetbrains.com/help/phpstorm/eslint.html
# package.json is required
npm init --yes

# Install eslint and corresponding standards into ./node_modules and generate .eslintrc.json file
npx eslint --init
38.4.2.15. npm Intellisense christian-kohler.npm-intellisense

Autocomplete npm modules in import statements based on package.json

38.4.2.16. Import Cost wix.vscode-import-cost
38.4.2.17. stylelint
38.4.2.18. Glitch for VS Code vs:code:ext:glitch.glitch
38.4.2.19. Sass syler.sass-indented

38.4.3. View

38.4.3.1. Editor Layout

38.4.4. Shortcuts

Command Pallete / Show All Commands
C-S-p or ⇧ ⌘ p
(no term)
Left bar is called View Bar that contains views
Explorer
C-S-e or ⇧ ⌘ e
Debug
C-S-d
Version control
C-S-g
Search
C-S-f
Extension
C-S-x
Toggle Side Bar Visibility
C-b
(no term)
View
Toggle Zen mode
C-k z
(no term)
Navigate Cursor
Go to Bracket (jump to open or end bracket)
C-S-\
Go to
C-g
Go to Symbol in File
C-S-o
(no term)
Navigate Files, Folders, Editors
Cycle through opened tabs
C-Tab quickopennavigateNextInEditorPicker
Close Editor
C-w
Reopen Closed Editor
C-S-t
Toggle Vertical/Horizontal Editor Layout
M-S-0
Split Editor
C-\
Go to File (search file)
C-p
Go to Definition
F12 or C-LeftClick
Go to Symbol in Workspace
C-t
Go Backward / Forward
M-Left M-Right
Find All References (in opened files only..)
M-S-F12
(no term)
Edit
Indent / Outdent
C-[ C-]
Format Selection
C-k C-f
Expand Selection
M-S-RightArrow
Move Line Up (move selection up)
M-UpArrow
Delete line
C-S-k
Trigger Suggest
C-Space
Toggle Block Comment
M-S-a
Toggle Line Comment
C-/
(no term)
Multiple cursors
Add Selection to Next Find Match
C-d, hit again will create another cursor which selects the same word
Select All Occurances of Find Match
C-S-l, select and create cursors for all matches in file
Create a cursor at caret
M-LeftClick
Add Cursor Above / below
C-M-Up
(no term)
Left or right to move the cursors
Right click on a variable name, Rename Symbol
F2
(no term)
Search
Search: Find in Files
C-S-f
While in view, see which files are included/excluded
C-S-j search.toggleQueryDetails
(no term)
Terminal
Toggle Integrated Terminal
C-`
Scroll Up (Line)
C-M-PageUp
Focus on Editor Group 1
C-1 View: Focus First Editor Group

38.4.5. Setting

Settings
C-,
User settings
saved in
Windows
~/.config/Code/User/settings.json or ~/AppData/Roaming/Code/User/settings.json
(no term)
Mac
User
~/Library/Application Support/Code/User/settings.json
Workspace
.vscode/settings.json at project folder
Open settings.json from VSCode
⌘ , select User or Workspace, and in the right top corner there's an icon to open
(no term)
Open the settings.json, C-S-p search Preferences: Configure Language Specific Settings
(no term)
Emmet
emmet.triggerExpansionOnTab
true
includeLanguages
{ "vue-html": "html", "mjml": "mjml" }
(no term)
Editor
showFoldingControls
always
wordWrap
on
formatOnType
true
formatOnPaste
true

38.5. WinMerge

  • Edit > Options
    • Backup Files
      • Create backup files into Global backup folder

38.6. WinSCP

https://winscp.net/eng/download.php

38.6.1. Connect to remote using SSH

  • New Session > New Site
  • SFTP
  • IP or full domain name, no alias
  • empty
  • Advanced > SSH Authentication > Private key file
    • it will generate if the private key is not Putty style e.g. id_rsa to id_rsa.ppk
  • Save the new site
  • Login!

38.6.2. Connect to remote using SSH with sudo su required

  • New Session > New Site
  • SCP
    Host name, Port Number, Username
    IP or full domain name, no alias
    Password
    empty
    (no term)

    Advanced

    • SSH Authentication > Private key file
    • it will generate if the private key is not Putty style e.g. id_rsa to id_rsa.ppk
      Environment > Directories > Remote directory
      change to /root
      SCP/Shell
      sudo su -
    (no term)
    Save the new site
    (no term)
    Login!
  • SFTP
    Host name, Port Number, Username
    IP or full domain name, no alias
    Password
    empty
    (no term)

    Advanced

    • SSH Authentication > Private key file
    • it will generate if the private key is not Putty style e.g. id_rsa to id_rsa.ppk
      Environment > Directories > Remote directory
      change to /root
      SFTP > Protocol options > SFTP server
      sudo su -c /usr/lib/sftp-server. Tested on AWS Ubuntu 16 and claimed working for 18
    (no term)
    Save the new site
    (no term)
    Login!

38.6.3. VS Code Extension alternative!

Extension name
SSH FS
(no term)
An icon will be created on the left nav bar
  • Setup an SSH connection. OpenSSH config cannot be pulled, so manual config is required
  • Extensions > this extension > setting icon

  • The settings.json is shared by VS Code as a whole, here's a sample of how 2 SSH FS hosts set up look like 

    {
    "emmet.triggerExpansionOnTab": true,
    "editor.showFoldingControls": "always",
    "editor.wordWrap": "on",
    "glitch.token": "1700535f-5133-41b1-81a2-7520deb26f75",
    "editor.fontFamily": "Source Code Pro",
    "editor.fontLigatures": true,
    "editor.fontSize": 13,
    "editor.formatOnType": true,
    "editor.formatOnPaste": true,
    "editor.renderControlCharacters": false,
    "editor.renderWhitespace": "all",
    "sshfs.configs": [
    {
        "name": "app1",
        "host": "1.2.3.5",
        "port": 22,
        "username": "nonrootuser",
        "privateKeyPath": "$HOME/.ssh/id_rsa",
        "sftpSudo": true
    },
    {
        "name": "app2",
        "host": "1.2.3.4",
        "username": "nonrootuser",
        "privateKeyPath": "$HOME/.ssh/id_rsa_2",
        "sftpSudo": true
    }
    ],
}

38.7. Regex

  • Regular expression is also called pattern. Syntax
    Boolean "or"
    |
    Grouping
    ()
    Quantification
    quantifiers
    • ?
    • *
    • +
    • e.g. {2} 2 times
    • {2,} minimum 2 times
    • {2,4} between 2 or 4 times
    (no term)
    Wildcard
  • Online Tester

38.8. Chrome

38.8.1. Resources

Blog (new releases)
https://developer.chrome.com/blog/

38.8.2. Command Menu (DevTools > C-S P)

Coverage (Show coverage)
JavaScript, CSS file unused bytes
Screenshot
full size screenshot

38.8.3. Toggle/detach dock side (DevTools > C-S D)

38.8.4. Navigate Panels (DevTools > C [ or C ])

38.8.5. Navigate among tabs ⌘ ⇧ a ⌥ ⌘ left/right

38.8.6. Console Panel

38.8.6.1. Console Commands

https://developers.google.com/web/tools/chrome-devtools/console/console-reference

  • console.count 

    function login(user) {
console.count("Login called for user " + user);
}

users = [ // by last name since we have too many Pauls.
'Irish',
'Bakaus',
'Kinlan'
];

users.forEach(function(element, index, array) {
login(element);
});

login(users[0]);
  • Formats DOM elements as JavaScript objects
  • can be an array, object or an array of objects
  • console.assert( condition, string ) Show string only when the first parameter evaluates to false console.assert(list.childNodes.length <= 500, "Node count is > 500");

  • Specifiers 

    //Style console output with CSS
console.log("%cThis will be formatted with large, blue text", "color: blue; font-size: x-large");

// format a value as string
console.log("%s has %d points", "Sam", 100);
console.group("Authenticating user '%s'", user);
console.group("Authorizing user '%s'", user);
console.log("User '%s' was authorized.", user);
console.groupEnd();
console.groupEnd();
    %i or %d
    integer
    %f
    float
    %o
    expandable DOM element as seen in Elements panel
    %O
    Formats the value as an expandable JavaScript object
    %c
    color and other styles
  • console.warn(), console.info(), console.error()
  • console.trace(object)
38.8.6.2. Command Line API
$('code') // Returns the first code element in the document
$$('figure') // Returns an array of all figure elements in the document
$x('html/body/p') // Returns an array of all paragraphs in the document body. XPath

$('[data-target="inspecting-dom-elements-example"]') // CSS selector

// $_ :: last evaluated expression
// Open the selected element on Elements Panel
inspect($_)

// $0, $1, $2, $3, $4 :: recently selected

// copy a JavaScript variable value, later paste to e.g. Notepad
copy(abc)

monitorEvents(document.body, "click");
38.8.6.3. Inject CSS file in Console

jQuery(document.head).append('<link rel="stylesheet" href="path_to_my_css">');

38.8.6.4. Offload CSS file in Console
document.styleSheets[0].disabled = true;

$0.disabled = true; // select the <link> or <style> element in Elements and then run in Console

38.8.7. Block Request or Domain - Network

Block a certain file request or any files from a domain. Network > right click on a file and Block Request

38.8.8. Get all events of an element in Console

getEventListeners($0);

38.8.9. XPath XPath

  • In DevTools > Elements, search using Xpath, e.g. find all h4 element //h4
  • xPath Online tools
  • select all nodes with name
  • select from the root node
  • select nodes from the current node that match the select no matter where they are
    //book
    select all book elements in the whole XML document
    .//book
    select all book elements which are under the current node
  • select the current node
  • select the parent of the current node
  • select attribute
  • first book element
  • last book element
  • second last book element
  • first 2 book elements
  • all book elements that have price element with a value greater than 35.00
  • select lang attributes of all title elements in book elements. Not filter!
  • match any element node
  • match any attribute node. Have at least one attribute
  • select multiple paths
38.8.9.1. Predicates
  • Things inside [] are called predicates
  • Multiple predicates
    • //*[local-name()='CSItem'][@name='NameAsReported'][@maxLength='250'], or
      • //*[local-name()='CSItem' and @name='NameAsReported' and @maxLength='250']
        • //*[local-name()='CSItem' and @name='NameAsReported' and @maxLength='250'][1]
          • Select the first CSItem element
        • //*[local-name()='CSItem' and @name='NameAsReported' and @maxLength='250'][1]/@description
          • Select the description attribute of the first CSItem element
    • //library/books/book[author[firstName[contains(text(),'Li')]]]
      //library/books/*[local-name()='book'][*[local-name()='author'][*[local-name()='firstName' and contains(text(), 'Li')]]]
      using local-name
      (no term)
      Select all book elements that that have author elements that have firstName elements that have text Li
    • //library/books/book/author/firstName[contains(text(),'Li')] or
      • //library/books/*[local-name()='book']/*[local-name()='author']/*[local-name()='firstName' and contains(text(), 'Li')]
      • Select all firstName elements (which has text Li) of the author elememts of the book elements
  • Functions
    contains()
    //author[firstName[contains(text(),'Li')]]
    text()
    text of a node
    (no term)
    starts-with()
    //author[starts-with(@firstName, 'Li')]
    attribute value starts with
38.8.9.2. Axis Syntax
  • axisname::nodetest[predicate]
  • axisname is the relationship name
    • self
    • child
  • nodetest is a node name to test
    child::book
    book nodes that are children of the current node
    attribute::lang
    lang attribute of the current node
    child::*
    children elements of the current node
    child::text()
    text node of the current node
    (no term)
    child::node()
38.8.9.3. XPath Operators
  • +, -, *, div
  • division remainder e.g. 5 mod 2
  • combine 2 node-sets
  • =, !=, <, <=, >, >=
  • price=9.8 or price=9.7
  • and

38.8.10. Turn on DevTools Experiment

chrome://flags > search a Feature, enable and relaunch

Developer Tools experiments :: Experiments under DevTools > Settings Experimental JavaScript

38.8.11. Elements Panel

38.8.11.1. Styles

Click on color and expand Contrast Ratio produces 1 line. Choose color below the line in order to make it AA or AAA compliant.

38.8.11.2. Accessibility

38.8.12. Audits Panel

Run checks using Light House

38.8.13. Sources Panel

38.8.13.1. Local Overrides

Specify a directory where DevTools should save changes. This directory will contain all changes across different website domains. DevTools > Sources > Overrides

Select folder > Allow > Enable Local Overrides

This only works to override styles that are loaded by a CSS file and you can override the CSS files.

38.8.13.2. Stop running script
  • For example, a JavaScript event (onmouseover) creates and displays a new div. We want to stop and investigate on the new div
  • This is different from adding :hover as it's for CSS only
  • Open DevTools, got to Sources panel, hover on the element, hit F8, move mouse to Sources panel, you will find the script is stopped running
    • Hit F8 again to resume script running
38.8.13.3. Event Listener Breakpoints and Blackboxing

Mouse > click Say it jumps jquery.js at function abc at line 1000. Find this under Call Stack, right click and Blackbox script. jquery.js will be ignored.

Right click again to stop blackboxing, To see all blackboxed scripts: DevTools > Settings > Blackboxing https://developer.chrome.com/devtools/docs/blackboxing

38.8.14. Extension

38.8.14.1. Google Tag Assistant chrome:google tag assistant
38.8.14.2. Postman
Raw request and response
Postman console (icon at the bottom)
POST key value is array
create multiple keys e.g.
  • key channel[0], value hello
  • key channel[1], value world
  1. Variable Scopes
    • https://learning.postman.com/docs/sending-requests/variables/
    • a workspace which has collections, requests, test scripts and environments
    • throughout the requests in a collection and are independent of environments
    • e.g. local vs testing or production. Only one environment can be active at a time
    • only in request scripts
    • come from external CSV and JSON files to define data sets
    pm.globals.set("global_variable_key", "variable_value");
pm.collectionVariables.set("collection_variable_key", "variable_value");
pm.environment.set("environment_variable_key", "variable_value");
pm.variables.set("local_variable_key", "variable_value");

pm.environment.unset("variable_key");

// {{local_variable_key}} in GUI

pm.variables.get("variable_key"); //access a variable at any scope including local
pm.globals.get("variable_key"); //access a global variable
pm.collectionVariables.get("variable_key"); //access a collection variable
pm.environment.get("variable_key"); //access an environment variable
    1. Dynamic Variables
  2. Tests tab for each request
  3. Pre-request Script for each request
    1. Moment.js

      const moment = require('moment');

// moment() === '2021-07-28T19:26:01.558Z' // UTC

pm.variables.set("mydate", moment().format("YYYY-MM-DD-HH")); // eq. to PHP Y-m-d-H. Time is converted to current timezone! 
// So if local timezone is America/Toronto then it's 2021-07-28-15

pm.sendRequest("http://worldtimeapi.org/api/timezone/America/Toronto", function (err, res) {
    pm.globals.set("localTime", res.json().datetime); // 2021-07-28T15:15:18.644592-04:00
});

var localTime = moment(pm.variables.get('localTime')).format("YYYY-MM-DD-HH"); // 2021-07-28-15

console.log(moment.utc().format("YYYY-MM-DD-HH")); // 2021-07-28-19
    2. CryptoJS 
      // no need to require

// eq. to PHP sha1(key)
var key = 'abc';
var hash = CryptoJS.SHA1(key);
console.info(hash.toString(CryptoJS.enc.Hex));
  4. Collection run
    • Setup Tests, Pre-script for each request
    • Select the requests and run them in sequence
  5. Pricing and Plans
    • Plans are for a Team
      • Free
        • Any sort of collaboration e.g. Collections for up to 3 users
      • Basics
      • Professional
      • Enterprise
  6. Team
    • An individual Postman account can create a Team and then the individual can pay for a subscription plan or request someone else to pay
      • Once the individual account (li@me.com) creates a Team (Li Team), the individual account becomes a Team account and all personal workspaces are under the Team account
        • To check if you have an individual account, click on the avatar on the top right, in between View Profile and Teams (above Settings), you will see li@me.com with subtitle Individual
      • li@me.com can join other Teams, but li@me.com still does not have an individual account
        • li@me.com now joins with Li Team (as a creator) and Li2 Team
      • When li@me.com leaves a Team (say Li2 Team) and li@me.com does not have an individual account, an option is provided to keep the personal workspaces of that Team in an individual account. Now li@me.com has an individual account
      • Now li@me.com leaves Li Team, because li@me.com now has an individual account, no option is provided to keep the personal workspaces of Li Team to the individual account. You will have to export data prior to leaving.
    • An individual Postman account can create a Workspace that has Visibility = Team
      • Visibility = Partner requires subscription plan Enterprise. It enables
      • And then the Workspace is available for members of the Team
38.8.14.3. Save Page WE

Chrome's default save page as HTML has some limitations such as not all images are downloaded. This saves all content in one HTML file.

Font file is not converted but you can fix the url() manually.

Turn off 38.8.24

38.8.14.4. Ignore X-Frame headers

Ignore headers X-Frame-x e.g. X-Frame-Options

38.8.14.5. Requestly

Modify http request and response headers

38.8.14.6. jQuery Audit

Find function that attaches events to a selected element. Element > jQuery Audit Expand the Events and RC on handler to Reveal function in source.

38.8.14.8. Wasp chrome:wasp
  • Web Analytics Solution Profiler
  • DevTools > beside Audits panel, there's a WASP panel
  • WASP tracks across page opening history (continously)
  • Scripts
    • Loaded JavaScript files
    • Show the path of JS files and may be cookies and headers that are set by the script
  • Tags
    • Tags are scripts that are recognized by WASP and it has deeper data
    • Tags that are triggered in GTM also appear here
    • googletagmanager.com
    • before googletagservices.com
    • google-analytics.com (analytics.js) ga:ga
    • googleadservices.com (old AdWords)
38.8.14.9. Hosts file

Host Switch Plus Virtual Hosts

38.8.14.10. ChroPath

Find XPath or CSS selector for an element. Go to DevTools

38.8.14.12. React Developer Tools
  • DevTools > tabs at top Components and Profiler
  • Extension Setting
    • Allow access to file URLs
  • Profiler tab
    • Recording needs to be turned on. Each event will be recorded separately e.g. a button is clicked twice and hence 2 profiling recordings
    • Setting
      • General
      • Profiler
        • Record why each component rendered while profiling

38.8.15. Network

38.8.15.1. Filter Post Requests

method:POST

38.8.15.2. Preserve log
  • Preserve all requests log even though the page URL is changed (e.g. after the Post request, URL changes)
    • Change of this setting will carry to all new DevTools sessions
  • For links that open in another tab, tell DevTools to auto open and hence all Network is tracked in the new tab/window
    • DevTools > F1 (Settings > Preferences > Global) > Auto-open DevTools for popups
    • If the parent tab does not have DevTools open, the new tab will not open the DevTools
    • You may need to enable Preserve Log in the parent tab first. But after that, whether or not Preserve Log on the parent tab does not matter.
38.8.15.3. Capture Screeshots

Double click to show image and single click to review timeline

38.8.15.4. Network Performance

Blue line is DOMContentLoaded and red line is load

In Waterfall for a request

Queueing
The browser queues requests when:
  • There are higher priority requests.
  • There are already six TCP connections open for this origin, which is the limit. Applies to HTTP/1.0 and HTTP/1.1 only.
  • The browser is briefly allocating space in the disk cache
Stalled
The request could be stalled for any of the reasons described in Queueing.
DNS Lookup
The browser is resolving the request's IP address.
Proxy negotiation
The browser is negotiating the request with a proxy server.
(no term)
Request sent. The request is being sent.
(no term)
ServiceWorker Preparation. The browser is starting up the service worker.
(no term)
Request to ServiceWorker. The request is being sent to the service worker.
(no term)
Waiting (TTFB). The browser is waiting for the first byte of a response. TTFB stands for Time To First Byte. This timing includes 1 round trip of latency and the time the server took to prepare the response.
(no term)
Content Download. The browser is receiving the response.
(no term)
Receiving Push. The browser is receiving data for this response via HTTP/2 Server Push.
(no term)
Reading Push. The browser is reading the local data previously received.

38.8.16. Change User Agent

F12 or C-S-i > 3 dots > More Tools > Network conditions > User agent > Chrome - Windows

38.8.17. Which DOM elements are repainted

F12 > 3 dots > More Tools > Rendering > Paint flashing

38.8.18. Performance

38.8.18.1. Memory
  • https://developer.chrome.com/docs/devtools/memory-problems/
  • Real time memory usage for a tab using Task Manager
    • 3 dots in the main window (not DevTools) > More Tools > Task Manager, right click on columns and show JavaScript Memory
  • Real time memory usage, CPU, number of DOM nodes and more using Performance monitor drawer
    • Drawer is in the Console Drawer
  • See Heap size by recording profiling in Time Line format in Performance Panel
    • DevTools > Command Menu > type Show Performance and go to Performance Panel > enable Memory and hit Start Profiling and reload page
      • Don't forget to hit the recycle button to force collect garbage when starting and ending a recording
  • See Heap Snapshots in Memory Panel
    • DevTools > Command Menu > type Show Memory and go to Memory Panel
38.8.18.2. Performance Insights Panel
  • ⌘ ⇧ p open Command Menu and type performance to open the Performance Insights panel
    • https://developer.chrome.com/docs/devtools/performance-insights/
    • Use W to zoom in, S to zoom out, A to go to left of the timeline, D to go to right
    • Can record or measure page load
    • Major events like Rendering blocking requests, long tasks and layout shifts are laid out in a vertical timeline. Those are usually the ones need special attention (insights) and details like call tree are available. Not all events/calls have details. For full details, record in the Performance Panel.
    • Key metrics in Timeline
      • First Contentful Paint (FCP)
      • DOM Content Loaded (DCL)
      • Largest Contentful Paint (LCP)
      • Cumulative Layout Shift (CLS)
38.8.18.3. Performance Panel
  • Command Menu > type performance to open Performance panel
  • https://developer.chrome.com/docs/devtools/performance/reference/
  • Don't forget to force collect garbage! See Chrome Monitor Memory
  • The first chart on top is called Overview and sections follow
    • Select a section and use W S A D to move the zoom in, move left, zoom out and move right in Overview.
      • Selected section or item will be outlined in blue.
      • Only the Main section shows JS Heap, Documents, Nodes, Listeners, GPU Memory
    • Sections
      • Network
        • HTML: Blue
        • CSS: Purple
        • JS: Yellow
        • Images: Green
    • Event, Function Call, Evaluate Script, Compile Script, etc.
    • Recalculate Style, Layout, Update Layer Tree, etc.
    • Under each section, the chart is called flame chart. Root activities are the ones at the top of Main section chart and they are activities that cause the browser to do some work.
  • Tabs
    • Call Tree
      • Which root activities cause the most work during the selected portion of a recording (range in Overview)
      • Columns
        Activity
        root activities followed by their children
        Self Time
        the time directly spent in that activity.
        Total Time
        the time spent in that activity or any of its children
      • On the top right, there's a button Show Heaviest Stack
        • shows which children of the selected activity took the longest time to execute
    • Bottom-Up
      • Which activities directly took up the most time in aggregate
    • Event Log
      • List activites in the order in which they occurred during the recording
38.8.18.4. Recorder Panel (experimental)
  • Command Menu > type Show Recorder and go to Recorder Panel
  • Start recording and perform user actions e.g. go to a page, click a button. Then stop recording
  • For each step, Add duration to make the step lasts longer (delay the time to go to next step)
  • Clicking Performance panel will perform the recorded and check out the Performance profiling recording!

38.8.19. Clear cache for one domain

Settings > Advanced settings > Content Settings under Privacy > All cookies and site data under Cookies > search a website and clear

38.8.20. Refresh Favicon

Delete files Favicons and Favicons-journal in this folder. It will remove all favicon cache C:\Users\your_username\AppData\Local\Google\Chrome\User Data\Default

38.8.21. Flush DNS Caches in Chrome

Even ipconfig /flushdns is run, Chrome still caches the DNS Navigate to chrome://net-internals/#dns and press "Clear host cache" button Or on the top right, down arrow > Tools > Clear Cache

Make sure bookmark has the right url. Say you bookmark abc.com as www.abc.com, if you type abc.com, Chrome will go to www.abc.com!

Also say before there was a DNS A record for www.abc.com to 1.2.3.4 which is an old IP. You remove it and set a CNAME for www.abc.com to abc.com, www.abc.com and https://www.abc.com will actually go to 1.2.3.4 which is not expected.

For more edge cases like above and in case of the local network DNS caches which you have no control of, set the network adapter to use 8.8.8.8 Google DNS.

38.8.22. Flash Couldn't Load Plugin

Properites > Security, give Full Control for Everyone in this folder C:\Users\you\AppData\Local\Google\Chrome\User Data\PepperFlash\some.version.number

38.8.23. Allow js:xmlhttprequest for local files

38.8.24. Allow CORS locally

  • May need to close all Chrome instances. Mine opens the Chromium version not the regular Chrome.
  • chrome.exe --disable-web-security --user-data-dir
  • open -n -a /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --args --user-data-dir="/tmp/chrome_dev_test" --disable-web-security
  • Errors such as
    • Access to Font at 'yourwebsite.com' from origin 'e.g. localhost' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'e.g. localhost' is therefore not allowed access.
  • A CORS request is an HTTP request that includes an Origin header. When do browsers send out CORS requests?

38.8.25. Remote Debug Android Devices

On Android, Settings > System > About phone > tap Build number 7 times, then Sytem > Developer options is enabled, enable USB debugging. It allows Android Studio and other SDK tools to recognize your device when connected via USB, so you can use the debugger and other tools. Other options on Android: https://developer.android.com/studio/debug/dev-options

Connect Android to test machine > DevTools > 3 dots > More tools > Remote devices > enable Discover USB devices.

When the phone is connected to a computer, on the phone say transfer photos and open the device on the computer so that the debug connection can be made.

On the phone, you may need to enable Always prompt when connecting to USB in Developer Options in order for prompting Use USB for Transfer Photos.

Don't let the phone screen turn off while debugging.

38.8.26. Debug

  • Add debugger; to JavaScript so that Chrome stops there as a breakpoint

  • Within an HTML <script> tag, set the name of the file so that it can be opened under Sources tab 

    <script>
    ... (your code here)
    //# sourceURL=somename.js
</script>

38.8.27. Command line

https://peter.sh/experiments/chromium-command-line-switches/ 

chrome.exe [option]...
chromium-browser [option]...

For headless

  • –headless
  • need to have it when CLI is running by a root user
  • By default, file:// URIs cannot read other file:// URIs e.g. js:xmlhttprequest for local files
  • –disable-web-security
    • Don't enforce the same-origin policy
    • See 38.8.24

38.8.28. Security chrome://settings/security

  • Enhanced protection vs Standard protection

38.9. Firefox

38.9.1. Make all new windows open in Private Mode

  • Visit about:config and click "Yes, I'll be careful"
  • Search for browser.privatebrowsing.autostart
  • Change value to True

38.10. BrowserStack

2018/03 Devices chosen for screenshots

iOS v8.3 iPad Mini 2, iPhone 6, iPhone 6 Plus v7 iPhone 5S, iPad Mini v6 iPad 3

Android Samsung S5, S4 Google Nexus 9, 7

Win 10 Edge 15, IE 11, Chrome 50

Win 8.1 IE 11

Win 8 IE 10

Win 7 IE 11, 10

Mac OS X 10.11 El Capitan Safari 9.1 Chrome 50

Mac OS X 10.10 Yosemite Safari 8 Chrome 50

Mac OS X 10.9 Mavericks Safari 7.1

38.11. Device Market share, Metrics

38.11.1. Browser

https://docs.google.com/spreadsheets/d/1iAOjjA-MG5nUiAidhP_MdmnM-kEbX1j6qpMI9zbKjCw/edit?usp=sharing

https://data.apteligent.com/ios/ iOS Overview 2018/03 <=6 .06% (.06%) <=7 .30% (.23%) <=8 .93% (.63%) <=9 6.5% (5.57%) <=10 20.5% (13.94%)

https://data.apteligent.com/android/ Android overview 2018/03 <=4 .41% 4.4 12%

https://netmarketshare.com Browser Market Share across all devices 2018/03 Chrome 58.57% Safari 17.15% Firefox 6.61% Internet Explorer 6.52% Edge 2.11%

Internet Explorer Browser Market Share across all devices 2018/03 IE 11 71.76% IE 10 5.5% IE 9 6.81% IE 8 12.19% IE 7 1.56%

http://gs.statcounter.com/os-version-market-share/windows/desktop/worldwide Windows Desktop 2018/03 Win 10 38.29% Win 7 44.58% Win 8.1 9.21% Win 8 2.5% Win XP 4.47% Win Vista .86%

http://gs.statcounter.com/macos-version-market-share/desktop/worldwide macOS Desktop overview 2018/03 High Sierra 10.13 14.75% Sierra 10.12 39.27% El Capitan 10.11 21.22% Yosemite 10.10 14.12% Mavericks 10.9 4.95% Mountain Lion 10.8 < 2.13% Lion 10.7 <2.13% Snow Leopard 10.6 2.13%

Resolution Distribution http://gs.statcounter.com/

38.11.2. Device Metrics tool:device metrics

https://material.io/tools/devices/
Device Platform, Screen dimensions in in or cm, aspect ratio, dp or dip (density-indepenndent pixels), px (CSS pixels), density (px/dp)
https://www.mydevice.io/
newer devices
http://screensiz.es/
By popularity
http://viewportsizes.com/
Old source. By release date
Device and viewport size in JavaScript
http://ryanve.com/lab/dimensions/
(no term)
iPhone X, iPhone 8 Plus, iPhone 7 Plus and iPhone 6s Plus has pixel density (scale factor) of 3x and others are 2x
Android devices
https://developer.android.com/about/dashboards/

38.11.3. Email Client Market Share

http://emailclientmarketshare.com/

38.12. Windows

38.12.1. Win Shortcuts

  • https://support.microsoft.com/en-ie/help/12445/windows-keyboard-shortcuts
  • switch input language and keyboard layout
  • minimize all windows except the active one
  • stretch active window to top while maintaining width
  • move active window to next monitor
  • App switching
    Win + number
    open the app # pinned to the taskbar. If the app is already running, switch to that app
    Win + Shift + number
    same as above but open a new instance
    Win + Ctrl + Shift + number
    same as above but open a new instance as Administrator
    Win + Ctrl + number
    switch to the last active window of the app pinned to the taskbar
    Win + Alt + number
    same as above but open the Jump List
    Win + T
    cycle through apps on taskbar
    Alt + Esc
    cycle through apps in the order in which they were opened
  • temp peak at desktop
  • display System Properties
  • open Windows Settings or go back to the current Windows Settings
  • enlarge text on display, enable night light mode
  • Win + P : presentation (change monitors)
  • Connect to other devices on network
  • Open Cortana in listening mode. Off by default. Start > Settings > Cortana and turn on "Let Cortana listen for my commands when I press the Windows logo key + C"
  • snipping, screenshot
  • Windows Whiteboard
    • Still need to use snip and then copy and paste to Whiteboard
  • File Explorer
    Ctrl + R vs F5
    refresh
    Alt + Left/Right
    Go back/forward
    Alt + Up
    Go up one folder level in File Explorer
    F3
    search in File Explorer
    Alt + Enter
    display properties for the selected item (right click menu)
    Shift + F10
    display shortcut menu for the selected item
    Ctrl + D vs Shift + delete
    delete and move it to recycle bin vs permanent delete
  • Inside a window or app
    F6
    cycle through screen elements in a window or desktop
    F10
    activate Menu in active app
    Alt + Spacebar
    open shortcut menu like Close, Maximize, Minimize the active app
    Ctrl + F4 vs Alt + F4
    close current document while the latter closes the whole app
    Ctrl + Y vs Ctrl + Z
    Redo and undo
  • Typing
    Win + .
    display Emoji
    Ctrl + Left/Right
    move cursor to the start or end of the next word
    Ctrl + Up/Down
    move cursor to the prev/next paragraph
  • resize Start menu when it's open
  • Remote Desktop Connection
    Swtich between host and remote
    C-M-Break then use M-Tab as usual to switch between apps on host

38.12.2. File Explorer

38.12.2.1. Search
(kind:pics OR kind:videos) AND size:>100KB
(*.jpg OR *.jpeg) AND size:>100KB AND (NOT tags:"to backup")

38.12.3. Win 7

Install KB3138612 and KB3145739 to boost Windows Update

38.12.4. Win 10

38.12.4.1. Boot to Safe Mode with Boot Options Menu

Start Menu > Power > hold Shift and click Restart Or use command line where you need to wait 1 minute shutdown.exe /r /o

38.12.4.2. Install a printer loop in "Type a printer name"

Run bcdedit.exe /set nointegritychecks on and restart Windows 10 to disable the driver signature enforcement Or Shift click on Power > Restart, choose Troubleshoot > Startup Settings > Restart, after restart choose F7 to disable driver signature enforcement

38.12.4.3. Windows version
  • CMD run winver returns Version number and OS Build Number
  • ver returns OS Build Number
  • Version number
    1607
    known as the Anniversary Update
    1703
    Windows 10 Creators Update
38.12.4.4. Upgrade from Windows 7

Make sure to do all critical and optional Windows Updates to ensure all drivers are up-to-date. Use Lenovo System Update to update all drivers and BIOS. If you experience black or blank screen during reboot to install Win 10, go to BIOS and change Display setting to Integrated (disable Discrete). Reboot and do the upgrade again.

Create Installation Media USB drive to install Win 10 on a different PC or reinstal

38.12.4.5. Ubuntu Bash on Windows

It's also called Windows Bash Shell Turn on Developer Mode (it's not necessary any more) You need to have Win 10 Anniverary Update, Creators Update is recommended. Make sure OS Build is not below 14393 Win + X > Settings > System > About Win + X > Settings > Update and Security > For developers > Developer Mode

Enable Windows Subsystem for Linux feature Search Windows Features > turn on Windows Subsystem for Linux (beta)

Win + S and type microsoft store, search linux and you will see different distros. Choose and install Ubuntu or a specific Ubuntu version. Then Win + S or go to Start > Dashboard to find the one just installed. Follow the username and password setup:

Reboot then open a command prompt, run bash and type y to install Ubuntu on Windows. If you see "Unsupported console settings", right click on the cmd top bar > Properties > uncheck Use legacy console

Create a UNIX user and password which have no relationship to Windows username and password The user will be added to sudo group and will signed-in automatically every Bash instance.

  • Multiple distro bash can be installed
    • List all installed Linux environments wslconfig /l
    • Set default Linux environment wslconfig /setdefault <name>
    • Run one of these to directly open a specific Linux environment ubuntu opensuse-42 sles-12
    • Run a command on a specific Linux environment in PowerShell or Command Prompt ubuntu -c apt-get moo
    • Run a command on the default Linux environment bash -c "uname -a"
  • Run wsl (or bash but deprecated) to get to Bash on Windows
  • Upgrade Linux Environment
  • Access Windows restricted folders

A Bash session with Windows admin privileges (run cmd as admin) may cd /mnt/c/Users/Administrator which is a directory restricted by Windows. While a Bash session without admin privileges would see Permission Denied.

Open File Explorer while you are in Bash for Windows
cmd.exe /c start .
(no term)
Run Windows command (e.g. to open IE) /mnt/c/Prgram\ Files\ \(x86\)/Internet\ Explorer/iexplore.exe
(no term)
Ubuntu is installed at %localappdata%\lxss\, don't modify files using Windows tools and apps here!
(no term)
Windows Drives, Files and File Permissions

Mount mnt/c or /mnt/e is drive C: and E:. Store files in any mount and modify files using Windows or Linux apps. It's not possible to modify file permissions under /mnt/(your_drive). scp from here to remote will have files/directories full permission! Change those on remote host after scp..

  • Remote files can be copied to any places other than /mnt/(your_drive). But don't open these files using Windows apps!

Permissions can be playe around in any places other than /mnt/(your_drive)

In order to connect to shared network drive, run this to create a d drive on Ubuntu and map it to Windows d drive 

sudo mkdir /mnt/d
sudo mount -t drvfs D: /mnt/d
38.12.4.6. WSL 2 
# Run as administrator Command Prompt
ver

# If Windows version is lower than 18917, then the old WSL 1 is being used.

# Update the WSL kernel
wsl --update

# list all images and WSL version
wsl --list --verbose

# eq.
wsl -l -v

# Set all images to run on WSL 2
wsl --set-default-version 2

# Inside WSL, find the host (Windows) IP
wsl.exe hostname -I
38.12.4.8. Startup Programs and Change HOMEDRIVE, HOMEPATH

Run shell:startup to find out the folder and copy shortucts of programs That will return the startup folder for your user account only. To find out the startup folder for all users, run shell:common startup

Active Directory might override some environment variables such as HOMEDRIVE HOMEPATH Create a batch file C:\Windows\System32\userinit.cmd 

@ECHO OFF
SET HOMEDRIVE=C:
SET HOMEPATH=\Users\%USERNAME%
SET HOMESHARE=\\localhost\C$\Users\%USERNAME%
@START C:\Windows\system32\userinit.exe

Set the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\Userinit to 'C:\Windows\System32\userinit.cmd,' Userinit default value is 'C:\Windows\system32\userinit.exe,'

38.12.4.9. Remove a keyboard

Win+S > Region & language settings > Select language and options > remove the keyboard if you see it, if not add the same language/keyboard and remove it.

Set default keyboard :: Control Panel > Language > Advanced settings > Override for default input method

38.12.4.10. Move a window

Shift + right click on a tab in Task Bar and you should see Move.

38.12.5. Win Server 2003

38.12.5.1. System Information

run winmsd.exe and look for processor to find out the bit count

38.12.5.2. Update_Windows.exe error

It consumes a lot of CPU and slows down the server. 2017/07/07. It's a hack.. Solution: Go to C:\Windows\Debug, Tools > Folder Options > View > Uncheck Hide Protected Operating System Files The batch files are referring to bitcoin mining website.. Ctrl+Alt+Del > Task Manager to stop Update_Windows.exe, then delete Update_Windows.exe, debug.exe, debug.bat, dll.bat

38.12.6. PowerSehll

38.12.6.1. Active directory
# see if a user `matt` is locked out
Get-ADUser matt -Properties * | Select-Object LockedOut
38.12.6.2. Get-Command
See file path of a command like linux which command
Get-Command ssh windows:c:which
38.12.6.3. Get-Command - gcm

Get manual of a command e.g. Select-String 

gcm Select-String | select -expand definition
38.12.6.4. Search text in files
Get-ChildItem -Path "C:\path" -recurse | Select-String -Pattern "find me" | group path | select -Unique name

# dir is an alias of Get-ChildItem, sls is an alias of Select-String
# Since -Path and -Pattern are the default parameters in Get-ChildItem and Select-String, they can be omitted
Get-ChildItem "c:\path" -recurse | Select-String "find me" | group path | select -unique name

# current directory is ".\"

# Exclude some binary files by file extension so that it runs faster
dir ".\" -recurse -Exclude *jpg, *jpeg, *png, *pdf | sls "<body>" | group path | select -unique name

# also exclude a folder, ? is alias of where
dir ".\" -recurse -Exclude *jpg, *jpeg, *png, *pdf | ?{ $_.FullName -NotLike "C:\fullpath\.git\*" } | sls "<body>" | group path | select -unique name

# exclude multiple sub folders
dir ".\" -recurse -Exclude *jpg, *jpeg, *png, *pdf, *.mp4, *.ogv, *.webm, *pdf, *.png, *.svg, *.otf, *.eot, *.woff, *.ttf | ?{ $_.FullName -NotLike "C:\fullpath\.git\*" -and $_.FullName -NotLike "C:\fullpath\abc\*" } | sls "<body>" | group path | select -unique name

# test dir without sls first to make sure you have the right "where" clause
38.12.6.5. Create script file

create C:\test.ps1 Run it 

& "C:\test.ps1"

Pass parameters, always use param in the first line in the relevant scope 

---- Begin script foo.ps1 ----
param([string]$foo = "foo", [string]$bar = "bar")
Write-Host "Arg: $foo"
Write-Host "Arg: $bar"
----  End script foo.ps1  ----
PS C:\> .\foo.ps1 -foo "foo" -bar "bar"
Arg: foo
Arg: bar

Function 

function foo([string]$foo = "foo", [string]$bar = "bar")
{
  Write-Host "Arg: $foo";
  Write-Host "Arg: $bar";
}

function foo()
{
  param([string]$foo = "foo", [string]$bar = "bar");
  Write-Host "Arg: $foo";
  Write-Host "Arg: $bar";
}

Scriptblocks 

PS C:\>& {param([string]$foo = "foo", [string]$bar = "bar") Write-Host "Arg: $foo"; Write-Host "Arg: $bar"; }
Arg: foo
Arg: bar
38.12.6.6. Shared network drive

Sometimes cmd and powershell do not recognize the shared network drive net use z: "\\remoteserver\subfolder"

38.12.6.7. Version

$PSVersiontable

38.12.6.8. Dracula Color Scheme
  • https://github.com/dracula/powershell
  • Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process
    In order to upgrade PowerShellGet
    Install-Module -Name PowerShellGet -Force
    (no term)
    Turns out have to lessen retristriction on policy permanently

38.12.7. Wireless hotspot

Tested with Win10 First, check if wireless adapter supports Hosted Network supported: Yes 

netsh wlan show drivers
# Create a wireless Hosted Network
netsh wlan set hostednetwork mode=allow ssid=WindowsCentral key="yourkey"
# Start the newly created Microsoft Hosted Virtual Adapter
netsh wlan start hostednetwork

WinKey + x and select Network Connections, select any network adapter (LAN or wireless) currently have an internet connection, Properties > Sharing > Allow ohter network users to connect through this computer's Internet connection > Home networking connection select the newly created "Local Area Connection* 13 (or other numbers)"

Start and stop the Microsoft Hosted Virtual Adapter 

netsh wlan stop hostednetwork
netsh wlan start hostednetwork
netsh wlan set hostednetwork mode=disallow
netsh wlan set hostednetwork mode=allow

Change setting 

netsh wlan set hostednetwork ssid=Your_New_SSID
netsh wlan set hostednetwork key="abc"
# View
netsh wlan show hostednetwork
netsh wlan show hostednetwork setting=security

It's not easy to delete the hostednetwork setting HKEY_LOCAL_MACHINE\system\currentcontrolset\services\wlansvc\parameters\hostednetworksettings Right-click the HostedNetworkSettings DWORD key, select Delete, and click Yes to confirm deletion. Reboot

38.12.8. SOCKS Proxy

Setup Windows to use SOCKS proxy for all internet connections. Win + s > Internet Options > Connections > LAN settings > Use a proxy server, go to Advanced, clear all protocols but config for SOCKS.

IP will be the proxy server IP but local DNS is used to resolve domain names to IP.

38.12.9. Windows Virtual PC

Install XP Mode on Windows 10 Pro and only Pro version can install

  • Google Windows XP Mode and download the en-us version
  • Enable Hyper-V in BIOS
  • In Win 10, run optionalfeatures.exe to go to Windows Features, enable all Hyper-V. Restart Windows
  • Control Panel > Admin Tools > Hyper-V manager
  • Download 7-Zip. Choose 64-bit if Win is 64.
  • Extract the downloaded XP mode file, go to folder Sources and find xpm file. right click and choose 7-zip > Open archive
  • Find VirtualXPVHD, extract it and rename to VirtualXPVHD.vhd file
  • Open Hyper-V Manager, go to Virtual Switch Manage to create a Virtual Switch
    • Type External, use default for others so that VM can use internet
  • In Hyper-V Manager, select the one and only local virtualization server on the left pane.
  • Action > New > Virtual Machine
    • Specify a name
    • Use Generation 1
    • RAM should 512MB or 1024MB
    • Select a Virutal Switch
    • Select an existing Virtual Hard Disk
    • Next, Finish
  • On the right pane in Hyper-V Manager, click Connect then Start
  • Set Region and Time
  • License is required..
  • Turn off Windows Update. Expect to see some driver installation failed
  • Most likely, internet doesn't work. Turn off VM.
    • Settings > Add Hardware > Legacy Network Adapter
    • Choose the Virtual Switch, Apply. Start VM

38.12.10. Find physical path by shared path

On server, run net share

38.12.11. Process Monitor procmon.exe

To troubleshoot a process, e.g. OUTLOOK.EXE Filter > Filter, or Ctrl+L, add Process Name is not OUTLOOK.exe Exclude, Apply Tools > Count Occurrences > Column: Result, double click on each Result to filter events

38.12.12. Dosbox

http://www.dosbox.com/download.php?main=1

This is to run old DOS programs. Double click to install to C:\DOSBox Put all your old DOS programs to, e.g. C:\OLDGAMES

Then go to cmd, run C:\DOSBox\dosbox.exe 

Z:\>MOUNT C C:\OLDGAMES
Z:\>C:
C:\>run your dos commands!

To automate the mount commands, double click on DOSBox-v-numberConfiguration, and a dosbox.conf text file pops up. Add these 2 under [autoexec] 

[autoexec]
MOUNT C C:\CT
C:

38.12.13. Batch Script

F7 in command prompt to review command history for the current window

38.12.13.1. Setting

Usually at the top of a .bat file. @ in front makes the command apply to itself only @echo off :: turn off output of the command itself

38.12.13.2. Comment

Rem Your comment in one line

38.12.14. FART windows:FART

  • Download
  • Replace text in files
  • fart.exe -i -r "C:\Dir\To\Files\*.txt" original_text new_text
  • fart.exe -i -r --remove *.txt "original text"
  • Say you want to remove double quotes. You will need to escape " with \"
  • fart -i -C --remove *.txt "\""
  • case insensitive
  • whole word
  • recursive
  • Find and replace filename instead of contents
  • Also find and replace in binary files
  • preview. no action

I realize fart sometimes cannot find files.. Use Linux

38.12.15. Windows Credential Store Panel Windows Credential Store Panel

User Accounts > Your Account > Manage your credentials Or Credential Manager Instead of Web Credentials, choose Windows Credentials

OneNote needs a password to sync this notebook. Delete any records MicrosoftOffice{15|16}_Data:Live:cid=some set of numbers Which have no username and restart OneNote.

38.12.16. MAC address

getmac /v /fo list

38.12.17. Meter an Ethernet Connection

Run regedit: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\NetworkList\DefaultMediaCost Right click on the key > Permissions > Advanced > Change next to Owner on the top Type Administrators and Check Names Check the box next to Replace owner on subcontainers and objects and click OK Back in the Permissions for DefaultMediaCost window, click Administrators to select the group Assign Full Control Allow, then OK Right click DefaultMediaCost > Ethernet and Modify Change from 1 to 2 for metered (DWORD 32-bit, Base Hexadecimal)

38.12.18. TS: "Installation Directory must be on a local hard drive"

Install .msi using admin privilege Shift+Right Click on the msi file and select copy path Open cmd run as admin and run: msiexec /i "the path"

38.12.19. TS: Win 10 random freezes

  • Download latest drivers
  • Win + R and type temp, delete all files
  • Advanced Power Plan > Hard Drive > Never turn off
  • BIOS turn off CPU C1E Function
  • Adjust Virtual Memory for C drive
    • Custom size: Initial size = Recommended, Max size = RAM * 1.5 in MB
    • Uncheck Automatically manage paging file size for all drives
  • Win + R and mdsched.exe to do a memory check
  • Check disk: right click on C drive > Properties > Tools > Check
  • Win + S, cmd, run as admin, sfc /scannow

38.13. Office 365

38.13.1. Exchange Online

38.13.1.1. Exchange Admin Center EAC ECP
  • New View of EAC admin.exchange.microsoft.com

    • Recipients

      Mailboxes
      does not include distribution list
      • UserMailbox, SharedMailbox
  • https://outlook.office.com/ecp/

    • Recipients

      mailboxes
      User Mail Box only
      groups
      Distribution List, Dynamic Distribution List, Office 365
      Office 365 Group
      collaborate between users both insdie and outside your company. Shared workspace for conversations, files and calendar events, and a Planner
      shared
      Shared Mail Box
  • ECP > permissions
  • ECP > compliance management

    • in-place eDiscovery & hold ms:ecp:in-place eDiscovery & hold

      • Use the new and better ms:scc:content search
      • Create a job with name, select all mailboxes or specific mailboxes, filter, set In-Place Hold settings, export to .pst file
      • In-Place Hold: to hold/preserve items matching the search query for a specific of time so that users or automatic detention policy cannot delete them. Requires Exchange Online Plan 2 or Exchange Online Archiving license for each mailbox
38.13.1.2. OWA
  • Outlook Web App. For O365 or Exchange Server 2016, it's called Outlook on the web
  1. Search

    https://support.office.com/en-us/article/search-mail-and-people-in-outlook-com-88108edf-028e-4306-b87e-7400bbb40aa7

    • Case insensitive
    • from to cc bcc participants
      • JerriFrye@contoso.com, JerriFrye, or "JerriFrye" can be used
      • from:"First Name Last Name"
      • to:"First Name Last Name"
    • body and subject
      subject:"Your Subject"
      phrase Your Subject
      subject:product plan
      "product" or "plan" in the subject
      subject:(product plan)
      with both product and plan in the subject
    • AND OR
      • from:"First Name Last Name" AND subject:"report" and from:"First Name Last Name"subject:"report" are equivalent
      • from:"First Name Last Name" OR subject:"report"
    • -from:"Jerri Frye"
    • sent received
      • only MM/DD/YYYY
      • sent:12/31/2019
    • hasattachment:yes hasattachment:no
    • isflagged:yes isflagged:no
  2. OWA Shared Mailbox
    • Shared mailbox doesn't require a license and quota is 50gb each. If it exceeds the quota, it will be locked
    • After it's added from Admin, user has to restart Outlook in order to see the shared mailbox
    • It takes one hour for new members to see the new shared mailbox
    1. Open Shared Mailbox in OWA
      • Click on your picture and select Open another mailbox
      • Type the shared mailbox name
    2. Add Alias and Move Messages
      • Add secondary email address to a shared mailbox and then setup Rules to that shared mailbox
      • Logon to OWA and open the shared mailbox primary email address
      • Gear > Options > Mail > Automatic processing > Inbox rules
      • Say the primary email is: abc@abc.com and a secondary email is: xyz@abc.com
      • You want to move emails sent to xyz@abc.com to a sub folder in the shared mailbox.
      • You need to use it includes these words in the message header = xyz@abc.com
    3. Add Members
      • A new member by default has full access:
        • Read, manage, and send as (send as the mailbox email address)
38.13.1.3. Message trace
  • Messages older than 90 days are unavailable. Filters are:
    • sender
    • recipient
    • sender original IP address
  • Url trace can only trace back 7 days
38.13.1.4. Microsoft Message Security & Compliance Center SCC ms:scc
  • https://protection.office.com/
  • Permissions: each child is called Role group name

    • eDiscovery Manager: perform searches and place holds on mailboxes, SharePoint Online sites, and OneDrive for Business locations

      • Roles
      • Export
      • RMS Decrypt
      • Custodian
      • Communication
      • Review
      • Preview
      • Compliance Search
      • Case Management
      • Hold
  • Search

    • Content search ms:scc:content search

      • Need to use Edge or IE to export search results
      • New interface compared to the old ms:ecp:in-place eDiscovery & hold
      • Assign yourself to ms:scc:eDiscovery Administrator
      • It can include shared mailboxes, OneDrive and other O365 account/product types
      • After creating a search, refresh, select and More > Export results and export as .pst file
      • O365 eDiscovery Export Tool needs to be installed
      • Copy and paste the key to download
  • Threat Management

    • Policy

      • Anti-spam
      • Default spam filter policy
        • Spam and bulk actions
          threshold
          7 (default)
        • Allow lists
          • Allow sender
          • Allow domain
        • Block lists
          • Block sender
          • Block domain
      • Connection filter policy
        • IP Allow List
        • IP Block List
38.13.1.5. Import PST Files
38.13.1.6. LDAP
  • Lightweight Directory Access Protocol
    • In a network, a directory tells where in the network something is located

    • An LDAP directory:

      • root directory
      • Countries

        • Organizations

          • Organizational Units: divisions, departments
          • Individuals: people, files and shared resources e.g. printers
    • An LDAP server is called a DSA (Directory System Agent)
      • An LDAP directory can be distributed among many DSA's with replication and synchronization
    • DNS (Domain Name System) is the directory system used to relate the domain name to a specific network address (each one is unique)
      • But you have to know the domain name
      • LDAP allows to search without knowing
    • LDAP is the protocol that Exchange uses to communicate with Active Directory

38.13.2. Bookings

  • Each Booking creates a user MyBookingName@contoso.com with Unlicensed

38.13.3. Teams

  • Channel General is created by default and members can add Conversation, Files or OneNote in a channel
    • Extra app can be added to a channel such as Stream, Website, Word, Excel etc.
  • Mobile app for M$ Teams
  • An org-wide team can be created and everyone will be added automatically

38.13.4. SharePoint

Site collection can have multiple subsites. Each site collection must have one top-level site. Defautl site collection and top-level site: yourcompany.sharepoint.com The default Team Site (yourcompany.sharepoint.com/TeamSite) is a top-site of the defautl site collection

See all site collections, Sharepoint admin center > site collections

Manage subsites and top-level site, go to one of the subsites and then click on Site Settings or go to the top-level site then Site Settings

38.13.4.1. Site Permissions

3 core default SharePoint permission groups: Owners, Members, Visitors.

You can add "Everyone except external" to a group.

Site Settings > People and Groups > Select the desired group > New

38.13.5. Office 2016

38.13.5.1. One-time-purchase Office License

Enter product key on a hard copy (card) on office.com/myaccount, get a digital product key.

To find out the digital product key of Office which is one-time-purchase.

Detailed Instructions

Office 2016 32-bit on Win 64-bit cscript "C:\Program Files (x86)\Microsoft Office\Office16\OSPP.VBS" /dstatus

Office 2016 64-bit on Win 64-bit cscript "C:\Program Files\Microsoft Office\Office16\OSPP.VBS" /dstatus

Office 2013 32-bit on Win 64-bit cscript "C:\Program Files (x86)\Microsoft Office\Office15\OSPP.VBS" /dstatus

Office 2013 64-bit on Win 64-bit cscript "C:\Program Files\Microsoft Office\Office15\OSPP.VBS" /dstatus

38.13.5.2. Outlook
  1. Export Contacts as CSV and Reimport to Outlook

    There's no option in Outlook for Mac to export contacts as CSV. OWA exports contacts without Company field. First, go to Outlook for Mac, select all contacts and drag them to a folder as VCF files. Open cmd on Windows, go to the folder with VCF files and run copy /B *.vcf all_in_one.vcf Use vCard to LDIF/CSV Converter to convert all_in_one.vcf to CSV Open the csv in Excel and Save as CSV so that file uses CR+LF Go to OWA, delete all contacts Always use PC Outlook to import CSV. OWA barely functions.

  2. Import Contacts

    Import from .pst file handles the field mapping automatically. Import from .csv you would need to Clear Map and Default Map.

  3. Rules: Multiple Senders

    Create a new contact folder and put contacts with emails. Create a rule which can be triggered when anyone in the contact folder belongs to "From" The rule is called "Sender is in specified Address Book"

  4. Rebuild OST

    Make a copy of the .ost file before delete. Then open Outlook again. If you can't copy/delete .ost file, close Outlook, Skype for Business and Office applications.

  5. Rebuild .srs file

    When rules don't work automatically but work manually, you should rename your-profile-name.srs at C:\Users\%username%\AppData\Roaming\Microsoft\Outlook

  6. Outlook for Mac
    1. Troubleshooting

      Sorry, we're having server problems, so we can't add Office 365 SharePoint right now. Please try again later.

      • Quit all Outlook and Office apps
      • Spotlight search and open KeyChain Access
      • Under Keychains > login on the right, delete all of these search results
        • Exchange
        • Office
        • ADAL
      • Launch Outlook
38.13.5.3. Excel
  1. Convert Seconds to dd:hh:mm:ss
    • Divide number of seconds by 86400, change cell Format to Custom and use dd:hh:mm:ss
38.13.5.4. OneNote

OneNote sync issue or remove crendentials. Refer to Windows Credential Store Panel

Move a OneOnte on SharePoint to personal OneDrive

  • Export the whole Notebook to a local .onepkg file
  • Open the .onepkg file and then File > Share > Move Notebook (you may need to Add a Place first)
  • Ctrl + M
38.13.5.5. Troubleshooting
  1. Outlook CSV Import

    A file error has occured in the Comma Separated Values translator while initializing a translator to build a field map. That's because the CSV file does not use CR+LF. Open it in Excel and Save As

38.14. Cmder

https://cmder.net/
it's called Commander Console
(no term)
Fork of ConEmu
(no term)
Just executable no installation needed even for the full version
  • Change the cmder.exe to run as administrator
(no term)
Preinstalled (full version)
  • Git for Windows

38.14.1. Shortcuts

Paste
S-Insert or C-v or right click or C-S-v
Copy from Cmder
S-click
Traverse up in directory structure
C-A-u
Zoom in, Zoom out
C-Mouse Wheel
Preferences / Settings
Win-A-p
History search
C-r
38.14.1.1. Tab / console
New tab
C-t
Close tab
C-w
Switch tab
C-Tab or C-S-Tab

38.14.2. Settings

  • Auto update is by default enabled
  • Features > Colors
    • Default Monokai

38.15. Reflector, ILSpy and Reflexil

C#, VB, F# and other .NET codes are compiled to Common Intermediate Language (CIL) or IL for short. Which is .dll file.

CIL is an object-oriented assembly language.

Red-gate .Net Reflector (trial) and ILSpy (open-source) can decompile a dll into source code.

Reflexil is a plugin for Reflect or ILSpy to change the OpCode for dll and export a new dll.

Reflexil v2.1 only works for ILSpy v2.4 but works for Reflector v9

Put Reflexil AIO (all-in-one version) dll to ILSpy.exe directory and run ILSpy.exe. View > Reflexil

Put Reflexil AIO dll to Reflector's Addins folder C:\Program Files (x86)\Red Gate\.NET Reflector\Desktop 9.0\Addins. Tools > Add-Ins, add the dll.

After edit, right click on the Object Browser > Reflexil > Save as

38.16. Project Management

38.16.1. Jira

38.16.1.1. Agile
  • 4 Values
    individuals and interactions
    over processes and tools
    • reduce multi-tasking
    working software
    over comprehensive documentation
    customer collaboration
    contract negotiation
    responding to change
    over following a plan
  • 12 Principles
  • Project manager's meeting responsibilities
    • fulfills an observer role
    • watches for unresolved issues
    • watches for roadblocks
    • ensures risks are decreasing over time
    • communicates status to stakeholders
  • Agile team size is less than 15 people
  • scrum's name. Also Kanban board
    Swimlanes
    User Stories, To Do, Doing, Done
  • Scrum vs Extreme Programming vs Kanban

    • Extreme programming

      • Embracing change vs Inspect and adapt (in Scrum)
      • User stories
      • Continuous integration
      • Test-driven development

      • Behavior-driven development

        • Scenario format
          scenario
          a user story
          given
          some set of initial conditions
          when
          an event/action occurs
          then
          an outcome is expected

        • Gherkin and Cucumber.io ./features/card_minimum.feature 

          # Comment
@tag
Feature: Card Minimum

    Scenario: Total charge is over the $2 credit card minimum
    Given Maria orders $3 of coffee from Li
    When Maria pays with a credit card
    Then Li should process the payment

    Scenario: Total charge is under the $2 credit card minimum
    Given Maria orders $1 of coffee from Li
    When Maria pays with a credit card
    Then Li should not process the payment
        • 5 values
      • Communication
      • Simplicity
      • Feedback
      • Courage
      • Respect

    • Kanban

      • Lean thinking
      • Respect for people
      • continuously improve
  • new, backlog and incomplete
    each feature on an index card is to meet a business need
    <action> <result> -> achieve a business objective
    • e.g. calculate tax for supplies ordered
    (no term)
    Group them by category and priority
    (no term)
    feature list reviewed > agreement by business > agreement by sponsor
    (no term)
    feature list is made and approved before each iteration starts
    (no term)
    estimate all features in terms of work at the beginning, and estimante relevant features before each iteration
    (no term)
    after the first estimate, create iteration, milestone and release plan
    (no term)
    In Speculate, decide which backlog or incomplete features should go in the current sprint
    (no term)
    Features (payment) > Themes (credit card payment) > Epics (visa payment) > Stories (actionable items.) > Agile estimation
    (no term)

    Define a feature by using use case

    • Actor (who) interacts with what or when what happens, actor does what
    • Performance Requirement Card
    • Feature ID: 123 - Invoices to Receive
    • Performance criteria: 90 days will be the max allowed time to receive invoices
    • Complexity factor: Medium
    • Acceptability point of measurement: 90 day review of received invoices to be done by receiving manager (who to or what is used to measure)
    • Verified by: Accounts receivable analyst
  1. Stages in Agile Life Cycle
    • Envision > Speculate > Explore > Adapt (may go back to speculate) > Close
      Envision

      what to build, resource and team you have, values & norms

      Project charter
      Scope, Objectives, Defined Stakeholders
      • Describes the customer's visions of the final product and overall boundaries for the project

      • Sample: University of Waterloo

        Vision Statement
        "Help people enjoy the outdoors and learn more about the wonderful birds that live on our planet"
        Mission Statement
        "Creating a cutting-edge technology platform that combines a website, a smartphone application, and location awareness to help our customers identify local birds"
        Success criteria
        "Sign up 10,000 users within 3 months of initial version"
        Product Data Sheet (PDS)
        1 to 3 pages
      • project description and high level project scope, usually taken straight from project charter
      • project objectives, the business value to provide
      • Timeline
      • Cost estimates
      • environmental, safety, economic (budget), technical (meet standards), political, project schedule (certain people are only available in November), team or product development
      • to balance (prioritize) scope, people, other resources, schedule and quality
        • Set up collaboration tools
        • e.g. work and meeting schedule, how they should cooperate
      • actively listen to what others are saying
      • attach the problem, not the person
      • seek to understand first
      • only focus on "this" sprint and feature
      • if you see a problem, say something and make suggestions for resolution
      • actively participate in the daily meetings
      • be engaged
      • solve conflict with the other pserson when possible
      • if not achievable, both parties must come forward for help from management
      • email is a low touch communication vehicle and should not be used to solve problems
      • don't respond to text messages during meetings
      • provide your full attention to the matter at hand
      • be respectful of one another
      • have the best interest of the project as your first priority
        • share and respect the roles and responsibilities of each team member
      (no term)

      Speculate, Explore and Adapt happen in each sprint or iteration

      sprint structure
      sprint size and keep it the same across the project
      • In each sprint, 1 week for speculate and 1 week for adapt
      • Determine sprint size by groups of features
      • small (20 hours), medium 40 and large 80
      (no term)
      Speculate
      • features based delivery plan and for each feature what estimates (time & resource) and risks are
      • Requirements broken down to features. At the end of each sprint, they will be implemented
      • if the sprint is 8 weeks, speculate phase is about 5 days
      (no term)

      Explore

      • Daily stand-up meetings. Each 15min or 30min max
      • what they achieved, what they will achieve today, anything they need help with
      • it's not the place to resolve issues
        • honest peer reviews of features among business team, testing and technical team
        • Identify roadblocks in daily work, collaboration, morale
      Adapt
      quick, often within one day
      • final review of the features by the customer
      • a documented meeting of team members to reflect on their performance (e.g. compare to plan, acknowledge member achievements)
      • discuss what is and is not working (brainstorm ideas to resolve issues)
      • agree to changes
      • so that lessons are captured and shared and future sprint plans are reviewed and adjusted
      (no term)
      Close
      • ensure all deliverables are completed
      • Retrospective (share wins, what did we do well? what could improve)
      • ensure vendors are paid and payments received
      (no term)
      Sprint Zero
      • Finalize the vision
      • purchase software
      • contract vendors
      • train
      • decide on Sprint duration
      • Release plan

    agile_life_cycle.png

  2. Scrum
    • 3 pillars
      • Transparency
      • Inspection
      • Adaptation
    • 5 Values
      • Focus
      • Respect
      • Openness
      • Courage
      • Commitment
    • Roles
      Product Owner

      who define what will be built and the order. It's called Customer Representative in Extreme Programming

      • Owns the product backlog, can cancel a sprint
      • Product Backlog > Release Backlog > Sprint Backlog
        • Requirements
        • Enhancement Requests
        • Defects
        • User Stories
        • New Features
        • PO solictis estimates on backlog items from team
      • 4-8 hours per week
      • detailed, estimated (Planning Poker), emergent, prioritized
      Development team member
      3 to 9 people
      • self-organized
      • a subset of product backlog
      (no term)
      Scrum master
      • coach, encourage and facilitate self-management
    • Events
      Sprint

      It's part of the Scrum framework. each takes 30 days or less

      sprint planning
      entire scrum team is required
      • first day of sprint
      • planning all sprint work
        • talk about new backlog items
        • discuss all backlog items

        • write tasks for each story

          • Tasking each story
          • Tasks or steps to take to meet acceptance criteria (AC)
          • Time-based estimate for each task
            • Add stories into Sprint Backlog for this sprint
            • Scrum master asks people to verbalize their commitments
        • plan the coming sprint, plus 2 more sprints
      • less than 2 hours or 8 hrs for a 30-day sprint
        • Daily Scrum
        • Sprint review ceremony, or Product Demo
      • listen feedback from other stakeholders after a sprint ends
      • less than 2 hrs or 4 hrs for 30-day sprint
        Sprint retrospective ceremony (Self-improvement)
        after sprint ends
      • formal 2-hour meeting or 3 hrs for 30-day sprint
      • Phases
        start
        create structure and safety
        data gathering

        extract info from the team in an organized manner

        • Starfish diagram
        • Keeping doing, less of, more of, stop doing, start doing
          • PANCAKE agenda
        • Puzzles
        • Appreciation
        • News
        • Challenges
        • Aspirations
        • Knowledge
        • Endorsements
        insights
        generate ideas about how to improve the team based on the data gathered
        decisions
        choose clear action items
        SMART
        specific, measurable, achievable, relevant, time-boxed
        closing
        gather feedback on the retrospective for next time
    • Artifacts
      • Product Backlog
      • Sprint Backlog
      • Product Increment
38.16.1.2. Story
  • In an agile framework, user stories are the smallest units of work
  • User stories are sketched out by the product owner, and then the entire product team collectively determines detailed requirements
  • As a {user role}, I want {goal} so that I {receive benefit/value}
    • Acceptance criteria (AC) is on the back of the card
      AC primary perspectives
      client, developer, tester
  • Evaluate if user stories are valid or good
    I
    indenpendently valued
    N
    negotiate with Product Owner to figure out the value each story should deliver
    V
    Customer Valuable
    E
    Estimable
    S
    Small. A user story should be delivered within a sprint
    T
    Testable
  • large grouping of work with common objective. How to split an epic into stories? Epic: I want to see a list of matching birds, so I can learn more about bird watching.
    F
    Flow. Story: As a bird finder, I want to see a list of matching birds on my smartphone, so I can learn more about birds when I'm away from my house
    E
    Effort
    • Story: As a bird finder, I want to see a list of matching birds sorted by likely matches, so I can learn more about bird watching
    E
    Entry of data
    • Story: As a bird finder, I want to see a list of matching birds based on my uploaded GPS coordinates, so I can learn more about bird watching
    D
    Data operations
    • Story: As a bird finder, I want to see an editable list of matching birds, so I can learn more about bird watching
    B
    Business rules
    • Story: As a bird finder, I want to see a list of matching birds based on the likelihood of a match on GPS coordinates and the image, so I can learn more about bird watching
    A
    Alternatives
    • Story: As a bird finder, I want to see a list of matching birds located in my zip code, so I can learn more about bird watching
    C
    Complexity
    • Story: As a bird finder, I want to see a list of matching birds based on color, GPS coordinates, and recorded autdio, so I can learn more about bird watching
    K
    Knowledge. Team has to do research and break it down into stories. Team create Story Spikes, they are not stories but the results of them can lead to stories
38.16.1.3. Feature
  • As a Fitness Club Manager, I should be able to set up a new club member account
    • It contains multiple workflows
38.16.1.4. Kanban board
  • Board Setting > Setup Kanban backlog
  • Assign status (To Do, In Progress, In Review, Done, etc.) to Kanban backlog
  • Number of cards (issues) can be controlled for each column (In Progress, Done)
  • Setup priorities of issues as swimlanes
  • Reports: Control Chart, Cumulative Flow Diagram
38.16.1.5. Gadgets

A dashboard contains gadgets which show stats of different projects

Average Age Chart
how long issues were unresolved
Sprint Health Gadget
% time elapsed, % of work donw, % of scope change
Jira Roadmap
# of resolved vs # of unresolved
Two Dimensional Filter Statistics
x and y axises
(no term)
Assigned to Me
Agile Wallboard
a small version of the current sprint scrum board
(no term)
Pie Chart
(no term)
Sprint Burndown
(no term)
Days Remaining
38.16.1.6. Add-ons
  • Settings > Add-ons > Find or Manage add-ons
  • Gadgets
    • Rich Text Gadget

    • Automation for Jira (paid)

      • Project rules
      • Close epic when all stories have been completed
      • Close stories when epic is completed
    • Planning Poker

    • Portfolio for Jira

      • To link epics and tasks across projects as dependencies using link type. Default link type is Block
      • e.g. Issue ABC-1 is blocked (as in Link other issues) by ABD-2 and ABE-3
        • To link releases across projects as dependencies using Plan which groups selected projects
      • Create cross-project release

38.16.2. YouTrack

38.16.2.1. Pricing
  • Based on number of users. The more users, the less cost per user
    • 1-3: free
    • 3-100: USD $4.16/user
    • 100-200: $3.33
    • 200-300: $2.5
    • 300-400+: $0.83
  • Buy more users for more storage: 1 user = 1 gb
38.16.2.2. Project
  • An issue can only have 1 project
    • Try to link another issue in other project. Another issue needs to be created in the other project
38.16.2.3. Default fields
  1. State
    • Settings > Custom Fields Settings > Fields List > State
      • See which values are considered as Resolved, others are Unresolved
    • Search
      • #resolved or #unresolved
  2. Assignee
    • May change to Can specify multiple values
    • Search
      • Refer to Login column in Users page for user name used in query
      • Assignee: li
      • include Li and Tom
      • exclude Li
      • exclude Li and Tom
      • me or unassigned
  3. Spent Time
    • Require Time Tracking is enabled in project
    • Search
      • Spent time: 1h .. 2h
      • greater than 1 day
      • empty value
38.16.2.4. Attribute-based Search
38.16.2.5. Reports
Time Report
https://www.jetbrains.com/help/youtrack/standalone/Time-Report.html
  • Total amount of time spent

  • Prerequisites

    • Enable time tracking in projects
    • Estimate field
    • e.g. Spent time
    • Work Item Type
      • e.g. Development, Testing, Documentation

  • Group by

    • Project
    • Work author
    • per project
  • Per
    • e.g. per user (work author), issue or work item
(no term)
Issues per project

  • Useful reports

    • Spent time percentage by project
    Projects
    include relevant projects
    Issue filter
    created: 2018-11-01 .. *
    Presentation
    Pie chart
    Show totals for
    Spent time
38.16.2.6. Mailbox Integration
  • Filter messages by patterns
    • Match message subject or text

38.16.3. Azure DevOps

38.17. Slack

https://api.slack.com/apps

38.17.1. Incoming Webhooks slack:incoming webhooks

38.17.2. Shortcuts

List all shortcuts
⌘ /
Move focus to the next section
F6, S-F6
  • ⌃ ⌘ left/right
Hover or select previous message in a channel
Up or Down
Add a reaction to this message
C-S-\ or Cmd-S-\
Edit this message
e
Go to a conversation
Cmd-k

38.18. Image

38.18.1. Sample Image

38.18.1.2. Text with Transparency

https://imgplaceholder.com/

First color is background color and second color is font color whose default is 757575 

https://imgplaceholder.com/420x320/transparent?text=hello
https://imgplaceholder.com/420x320/transparent/fff?text=hello

https://imgplaceholder.com/420x420/ca5353/757575?text=hello+world
https://imgplaceholder.com/420x420/ca5353?text=hello+world
https://imgplaceholder.com/420x420/ca5353?text=hello+world_br_Im+here+with+a+line+break
https://imgplaceholder.com/420x420/ca5353?text=hello+world_br_Im+here+with+a+line+break&font-size=12

https://imgplaceholder.com/420x420/ca5353?text=hello+world&font-size=12&font-family=Roboto

https://imgplaceholder.com/420x320/b13939/fff/glyphicon-tags
https://imgplaceholder.com/420x320/transparent/fff/glyphicon-tags

https://imgplaceholder.com/420x320/transparent/fff/ion-android-more-vertical

https://imgplaceholder.com/420x320/b13939/fff/fa-android
38.18.1.3. Real Image - Picsum.photos

Pick a picture that you like :: https://picsum.photos/images https://picsum.photos/

Specific picture :: https://picsum.photos/200/300?image=0

Random picture :: https://picsum.photos/200/300/?random

Blur :: https://picsum.photos/200/300/?blur

Grayscale :: https://picsum.photos/g/200/300

Square :: https://picsum.photos/200

38.18.1.4. Free PNG with transparency

http://www.stickpng.com/

38.18.2. Convert image to favicon (.ico)

38.18.3. WebP

Supported by Chrome desktop and android and Opera.

https://developers.google.com/speed/webp/faq#how_can_i_detect_browser_support_for_webp Detect if request header accept literally contains image/webp. Browsers have these accept headers 

*/*
image/*

Some CDN support content negotiation in action means if the accept doesn't have image/webp then deliver .png or .jpg. How to deliver different cache based on the same URL? Nginx example 

server {
   listen 80;
   server_name ei8gd7lwnymbl2d.cdnconnect.netdna-cdn.com jdorfman.cdnconnect.com;
   set $webp "";
   if ($http_accept ~* image/webp) {
       set $webp "webp";
   }
   location ~ /purge(/.*) {
       allow 192.168.0.1/24;
       deny all;
       proxy_cache_purge my_diskcached ei8gd7lwnymbl2d.cdnconnect10$myae$webp$1$is_args$args;
   }
   location / {
     [clipped]
       proxy_cache_key ei8gd7lwnymbl2d.cdnconnect10$myae$webp$uri$is_args$args;
   }
}

Notice how the “$webp” variable gives us different cache keys for the various versions.

More on WebP and Nginx :: https://www.igvita.com/2013/05/01/deploying-webp-via-accept-content-negotiation/

Use html:picture modernizr:webp

38.19. cPanel WHM

38.19.1. WHM Auto Backup

Do it using WHM and don't do it using cPanel as it needs to setup scripts to do backup on cPanel search Backup Configuration > Enable https://www.inmotionhosting.com/support/product-guides/dedicated-hosting/setup-scheduled-cpanel-backups

Default backup directory is /backup

38.19.2. WHM/cPanel DNS Record Files

Directory /var/named/yourdomain.com.db

Create full backup and inside folder dnszones, there's yourdomain.com.db

38.19.3. WHM Add IP to Firewall - Permit SSH Access

In order to see the list of IPs, cat /etc/apf/allow_hosts.rules

38.19.4. WHM List of Accounts

Modify Account

Privileges
Shell Access (SSH)

38.19.5. WHM Security Advisor

38.19.6. WHM Access

https://example.com:2087 http://example.com/whm (port 2086) http://whm.example.com (port 80)

38.19.7. SSH access

For VPS account, you might need to add IP to allowed rule under "Add IP to Firewall".

38.19.8. autossl:whm

38.19.9. Remote MySQL connection - cPanel

Remote MySQL and add remote IP address. Use the cPanel domain for db host https://www.inmotionhosting.com/support/website/databases/setting-up-a-remote-mysql-connection-in-cpanel

38.19.10. Access Log - cPanel

Metrics > Raw Access. Current logs :: ~/access_logs Archived logs :: ~/logs

38.19.11. Root domain A record change - cPanel

You want to point the root domain to antoher server. Assuming root domain originally has A record 1.2.3.4 and new IP is 2.2.3.4

  • Change root domain A record to 2.2.3.4
  • Change mail.yourdomain.com CNAME yourdomain.com to A record 1.2.3.4
  • Change smtp.yourdomain.com CNAME yourdomain.com to CNAME mail.yourdomain.com
  • Change yourdomain.com MX yourdomain.com to mail.yourdomain.com

38.20. Web Hosting

38.20.1. Comparison

hosting:pantheon is good for hosting one website

Charged by monthly pageviews

features $25 $100 $400
pageviews 10k 100k 500k
storage gb 5 20 30
memory 256 256 512
PHP workers 4 8 16
database working set 128 500 1gb
       

hosting:wp-engine is good for 1, 5 and 15 websites

Charged by monthly visits and bandwidth. Multisite is possible.

features $35 $115 $290
sites 1 5 15
visits 25k 100k 400k
bandwidth gb 50 200 400
storage gb 10 20 30
       

Digital Ocean has no bandwidth cap.

38.20.2. Wix

  • Trial starts on 2021/10/07 for 14 days
38.20.2.1. Editor
  • Dev Mode > Specific Page Code
    • Page code is run when that specific page is visited
    • The master JS file beside the Page Code is run whenever any page is visited
38.20.2.2. Settings
  • Domains

38.21. Tech Stack

38.22. Website Status

StatusCake - Free for 10 websites, 5 minutes check, random test locations Pingdom - Free for 1 website only.

38.23. Website Security, Testing

38.23.1. XSS

Tests 

http://a.ca/abc-xyz--><form><button formaction='/esi.attack.sim'><!--
http://a.ca/abc-xyz--><script src='a' onerror=alert(document.domain)>

38.24. Webhook

  • Webhook.site
  • ngrok.com
    • Host your local http app on the internet
    • See 17.17.27

38.25. Send password

38.26. RESTful API

38.27. Screencast + Audio, FFmpeg

38.27.1. ShareX

ShareX Download the portable version, Task Settings > Capture > Screen recorder > Screen recording options… Click Download to download the latest FFmpeg.exe Under Sources, hit Refresh until you see the right Audio source. Then go back to UI, Capture > Screen recording

38.27.2. OBS Studio linux:app:obs-studio

38.27.4. ffmpeg

  • Convert a gif to mp4
    -pix_fmt yuv420p
    MP4 store pixels in different formats and this is the max compatible across all browsers
    -vf

    MP4 videos using H.264 need to have a dimensions that are divisible by 2 

    ffmpeg.exe -i animated.gif -movflags faststart -pix_fmt yuv420p -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" video.mp4

      <img src="video.gif" alt="" width="400" height="300" />

  <video autoplay="autoplay" loop="loop" width="400" height="300">
<source src="video.mp4" type="video/mp4" />
<img src="video.gif" width="400" height="300" />
  </video>
  • Refer to youtube:encoding

38.29. Video

38.29.1. JW Player video:jw

  • Started as an open source used by YouTube before it's acquired by Google. Now ti's proprietary and used by ESPN, Electronic Arts and AT&T
  • Pricing
    • Custom Enterprise pricing, bills every 3 months

38.29.2. Brightcove video:brightcove

  • Public company
  • Pricing
    • Custom pricing
  • Migration
    • Looks like there's no way to migrate a sub account (its videos, playlists, players, stats) to another master account

38.29.3. Telaria

  • Video Management Platform (VMP)
    • Video advertising
  • previously called SlimCut
    Features

    https://outstream.telaria.com/

    • Formats
    • Carousel, Image + Copy, Vertical Video, Outstream, Video + Copy, Leave Behind, Footer Banner
    (no term)

    Cost

    • Direct sales on desktop/tablet
    • CAD $2.5 CPM on the first 4M impressions monthly
    • CAD $1.5 CPM in between 4M and 15M impressions monthly
    • CAD $1 CPM for over 15M impressions monthly
      • Direct sales on mobile
    • Addtional CAD $0.5 CPM added on top of desktop/tablet
      • Third party tags on desktop/tablet
    • CAD $2.5 CPM on the first 4M imp monthly
    • CAD $1.5 CPM in between 4M and 15M imp monthly
    • CAD $1 CPM for over 15M imp monthly
      • Third party tags on mobile
    • Addtional CAD $0.5 CPM added on top of desktop/tablet

38.29.4. YouTube Download

http://www.clipconverter.cc

38.29.5. 猫抓

][Chrome Extension]]

  • Get the m3u8 link
    • Method 1: Play the video and search m3u8 in Network on Chrome
    • Method 2: Open 猫抓, grab the link
  • Open 猫抓 > 媒体控制/其他功能 > M3U8解析器 > paste the link > 解析 > 调m3u8DL下载 > 合并下载

38.29.6. Kodi

38.29.6.1. Settings
  • Display Chinese characters, you need to go to

Settings > Appearance > Fonts > Arial based

  • Add apps to Homepage e.g. Video

Settings > Appearance > Settings > Add-on

38.29.6.2. File Manager

Install from zip file Folder xbmc-repos > english >

  • repository.exodus-1.0.1.zip (Exodus repository)
  • repository.xbmchub-1.0.6.zip (TVADDONS.ag Addon Repository)
  • chinese-repository.kodi-addons-chinese-1.0.2.zip (Kodi Add-ons of Chinese)
  • chinese-repository.kodi-repo.zip (Joname's repo)
  • chinese-repository.xbmc-addons-chinese-1.2.0.zip (Chinese Add-ons repository)

Install from zip file Folder isengard > all > superrepo.kodi.isengard.all-0.7.04.zip (Superrepo All repository)

38.29.6.3. Addons

Video Exodus (Exodus Artwork) Use Torba, Putlocker as the video sources (Exodus repository) Phoenix (TVADDONS.ag Addon Repository) icdrama.se (Superrepo All repository) azdrama.net (Superrepo All repository)

Subtitle 163sub (Chinese Add-ons repository) Sub HD (Chinese Add-ons repository) subom (Chinese Add-ons repository)

zimuku (Chinese Add-ons repository)

38.30. Audio

38.30.1. SoundCloud audio:soundcloud

  • Plan
    • SoundCloud Basic (Free)
      • 3h upload time
    • SoundCloud Pro Unlimited (CAD $15/month)
      • Unlimited upload time
      • Scheduled releases
    • Limit for all plans
      • single sound file < 4GB and time per track < 6hr45min
  • Migration
    • Cannot merge accounts
    • One email address one SoundCloud account

38.30.2. LibSyn

  • Public company
  • Features
    • IAB v2.0 certified stats: user agent, geographic, download destinations (different podcast platforms), social media, day/week/month, date range, episode breakdown, report downloading
    • Publish to podcast platforms using Libsyn RSS Feed and OnPublish
      • Apple Podcast
      • Spotify
      • Google Podcasts
      • Pandora
    • WordPress Libsyn Publisher Hub
    • Publish Audio, Video, PDF and Text Posts

38.31. Google

38.31.1. Google Public DNS

8.8.8.8 8.8.4.4

38.31.2. Google S2

Get favicon of any website http://www.google.com/s2/favicons?domain=http://superuser.com/

38.31.3. Google reCaptcha

  • https://developers.google.com/recaptcha/intro
  • allows you to verify if an interaction is legitimate without any user interaction. It is a pure JavaScript API returning a score, giving you the ability to take action in the context of your site: for instance requiring additional factors of authentication, sending a post to moderation, or throttling bots that may be scraping content
  • An owner can remove another owner
38.31.3.1. Client - Basic
<html>
  <head>
    <title>reCAPTCHA demo: Simple page</title>
     <script src="https://www.google.com/recaptcha/api.js" async defer></script>
  </head>
  <body>
    <form action="?" method="POST">
  <!-- has to have class="g-recaptcha" -->
  <div class="g-recaptcha" data-sitekey="your_site_key"></div>
  <br/>
  <input type="submit" value="Submit">
    </form>
  </body>
</html>
38.31.3.2. Client - Defer loading, explict rendering one widget/instance
<script type="text/javascript">
  var onloadCallback = function() {
    alert("grecaptcha is ready!");
    grecaptcha.render('html_element', {
  'sitekey' : 'your_site_key'
    });
  };
</script>

<form action="?" method="POST">
  <div id="html_element"></div>
  <br>
  <input type="submit" value="Submit">
</form>

<script src="https://www.google.com/recaptcha/api.js?onload=onloadCallback&render=explicit"
    async defer>
</script>
  1. Use namespace
    // v2 2020-10-22
myNamespace = {
  myRender: function() {
    grecaptcha.render('html_element', {
  'sitekey' : '...'
    });
  },
  loadReCaptcha: function() {
    var g = document.createElement('script'), s = document.scripts[0];
    g.src = 'https://www.google.com/recaptcha/api.js?render=explicit';
    // cannot use dots in onload e.g. myNamespace.myRender
    // https://www.google.com/recaptcha/api.js?onload=myNamespace.myRender&render=explicit
    g.async = true;
    g.defer = true;
    g.onload = function() {
  // this hack does not guarantee grecaptcha.render is loaded
  // have to wrap with grecaptcha.ready
  grecaptcha.ready(function() {
    grecaptcha.render('html_element', {
      'sitekey' : '...'
    });
  });
    };
    s.parentNode.insertBefore(g, s);
  },
};

myNamespace.loadReCaptcha();
38.31.3.3. Client - Multiple reCaptcha widgets/instance
<html>
  <head>
    <title>reCAPTCHA demo: Explicit render for multiple widgets</title>
    <script type="text/javascript">
  var verifyCallback = function(response) {
    alert(response);
  };
  var widgetId1;
  var widgetId2;
  var onloadCallback = function() {
    // Renders the HTML element with id 'example1' as a reCAPTCHA widget.
    // The id of the reCAPTCHA widget is assigned to 'widgetId1'.
    widgetId1 = grecaptcha.render('example1', {
      'sitekey' : 'your_site_key',
      'theme' : 'light'
    });
    widgetId2 = grecaptcha.render(document.getElementById('example2'), {
      'sitekey' : 'your_site_key'
    });
    grecaptcha.render('example3', {
      'sitekey' : 'your_site_key',
      'callback' : verifyCallback,
      'theme' : 'dark'
    });
  };
    </script>
  </head>
  <body>
    <!-- The g-recaptcha-response string displays in an alert message upon submit. -->
    <form action="javascript:alert(grecaptcha.getResponse(widgetId1));">
  <div id="example1"></div>
  <br>
  <input type="submit" value="getResponse">
    </form>
    <br>
    <!-- Resets reCAPTCHA widgetId2 upon submit. -->
    <form action="javascript:grecaptcha.reset(widgetId2);">
  <div id="example2"></div>
  <br>
  <input type="submit" value="reset">
    </form>
    <br>
    <!-- POSTs back to the page's URL upon submit with a g-recaptcha-response POST parameter. -->
    <form action="?" method="POST">
  <div id="example3"></div>
  <br>
  <input type="submit" value="Submit">
    </form>
    <script src="https://www.google.com/recaptcha/api.js?onload=onloadCallback&render=explicit"
    async defer>
    </script>
  </body>
</html>
38.31.3.4. api.js options
onload   Optional. The name of your callback function to be executed once all the dependencies have loaded.
render explict onload Optional. Whether to render the widget explicitly.
    Defaults to onload, which will render the widget in the first g-recaptcha tag it finds.
hl See language codes Optional. Forces the widget to render in a specific language.
    Auto-detects the user's language if unspecified.
38.31.3.5. class="g-recaptcha" tag attributes
tag attribute grecaptcha.render parameter Value Default Description
data-sitekey sitekey     Your sitekey.
data-theme theme dark light Optional. Color
    light    
data-type type audio image Optional.
    image    
data-size size compact normal Optional.
    normal    
data-tabindex tabindex   0 Optional.
data-callback callback     Optional.
        To execute when the user submits
        a successful response: g-recaptcha-response
data-expired-callback expired-callback     Optional.
        To execute when recaptcha response expires
        and the user needs to solve a new CAPTCHA
38.31.3.6. JavaScript API
  1. grecaptcha.render(container,parameters)

    Renders the container as a reCAPTCHA widget and returns the ID of the newly created widget. container :: The HTML element to render the reCAPTCHA widget. Specify either the ID of the container (string) or the DOM element itself. parameters :: An object containing parameters as key=value pairs, for example, {"sitekey": "your_site_key", "theme": "light"}. See grecaptcha.render parameters.

  2. grecaptcha.reset(opt_widget_id)

    Resets the reCAPTCHA widget. opt_widget_id Optional widget ID, defaults to the first widget created if unspecified.

  3. grecaptcha.getResponse(opt_widget_id)

    Gets the response for the reCAPTCHA widget. opt_widget_id Optional widget ID, defaults to the first widget created if unspecified.

38.31.3.7. Server Validation
  • For web users, a new field (g-recaptcha-response) will be populated in HTML and you can get the user’s response in one of three ways
    • A new form field with id g-recaptcha-response is created inside the container
      • POST parameter when the user submits the form on your site
    • grecaptcha.getResponse(opt_widget_id) after the user completes the CAPTCHA challenge
    • As a string argument to your callback function if data-callback is specified in either the g-recaptcha tag attribute or the callback parameter in the grecaptcha.render method
  • URL: https://www.google.com/recaptcha/api/siteverify
  • METHOD: POST
  • Parameters
    secret
    Required. The shared key between your site and reCAPTCHA
    response
    Required. The user response token provided by reCAPTCHA, verifying the user on your site
    remoteip
    Optional. The user's IP address

API Response 

{
  "success": true|false,
  "challenge_ts": timestamp,  // timestamp of the challenge load (ISO format yyyy-MM-dd'T'HH:mm:ssZZ)
  "hostname": string,         // the hostname of the site where the reCAPTCHA was solved
  "error-codes": [...]        // optional
}

Error codes missing-input-secret :: The secret parameter is missing. invalid-input-secret :: The secret parameter is invalid or malformed. missing-input-response :: The response parameter is missing. invalid-input-response :: The response parameter is invalid or malformed. bad-request :: The request is invalid or malformed. 

private function validateReCaptcha() {
  if (!isset($_POST['g-recaptcha-response']))
    return FALSE;
  $C_CAPTCHA                = $_POST['g-recaptcha-response'];
  $C_CAPTCHA_google_request = [
    'url'     => 'https://www.google.com/recaptcha/api/siteverify',
    'options' => [
  'secret'   => 'your-key',
  'response' => $C_CAPTCHA,
            // 'remoteip' => $_SERVER['REMOTE_ADDR'], // optional
    ],
  ];

  $ch      = curl_init();
  $options = [
    CURLOPT_URL            => $C_CAPTCHA_google_request['url'],
    CURLOPT_POSTFIELDS     => http_build_query($C_CAPTCHA_google_request['options']),
    CURLOPT_FOLLOWLOCATION => 1,
    // CURLOPT_HEADER         => 0, // not to include header in response
    CURLOPT_RETURNTRANSFER => 1,
        // CURLOPT_SSL_VERIFYPEER => 0,
        // CURLOPT_SSL_VERIFYHOST => 0,
        // CURLOPT_POST => 1,
  ];
  curl_setopt_array($ch, $options);
  $C_CAPTCHA_google_result = json_decode(curl_exec($ch), TRUE);
  curl_close($ch);

  if (is_array($C_CAPTCHA_google_result) && $C_CAPTCHA_google_result['success'] === TRUE) {
    return TRUE;
  }
}
38.31.3.8. CSS
/* center v2 2020-10-22 */
.g-recaptcha > div {margin:0 auto;}

/* The button might be too large for width < 320px */
@media (max-width: 320px) {
  .g-recaptcha {
    transform:scale(0.77);
    -webkit-transform:scale(0.77);
    transform-origin:0 0;
    -webkit-transform-origin:0 0;
  }
}

38.31.4. Google Search - Advanced

https://bynd.com/news-ideas/google-advanced-search-comprehensive-list-google-search-operators/

https://support.google.com/websearch/answer/2466433

Search operators: google:search operators 

site:nytimes.com
site:.gov
@twitter
camera $400
#throwbackthursday
jaguar speed -car
"tallest building"
largest * in the world
$50..$100
marathon OR race
related:time.com
# get details about a site
info:time.com
cache:time.com

Google Search usually ignores punctuation that isn’t part of a search operator.

38.31.5. Google Ads Personalization Setting

https://www.google.com/settings/ads

38.31.6. Google Partners

First, set up your company as a Google Partners.

  • Use a non-gmail email address to create an account to google.com/partners e.g. abc@yourcompany.com
  • abc@yourcompany.com has to have edit or admin access to an AdWords account. This AdWords account can be registered using any Google account. Login to that AdWords account and add abc@yourcompany.com with edit or admin access.
  • Once you created one Partner Profile account, go to Overview > My Profile, fill up address and phone number and select AdWords account as the "Top-level AdWords manager account"
  • Go back to Overview, and add your company

After that, any Google account users without @yourcompany.com can affiliate with that company. However, an individual can only affiliate with one company.

  • Sign in as abc@yourcompany.com to Google Partners and go to Overview > My Company > People to approve affiliation requests.

An Individual can work with who ever they wish, but can only affiliate their credentials of passing the required exams to ONE Company.

  • An individual is now required to have admin, standard or read-only access to the company's AdWords manager account.

Google Partner is set up in basically for Companies that meet certain requirements of Certification, Ad Spend, and Best Practices qualify and earn the GOOGLE PARTNERS BADGE.

The benefit is to the company to have your credential of passing the exams on their roster, or towards them fulfilling the TEST requirements to become a Google Partner

Individuals who receive certification but are not considered Google Partners unless they meet the other requirements of Best Practices (from their linked MCC account) and Ad Spend levels.

38.31.7. Gmail

38.31.7.1. Use Gmail SMTP to send emails
38.31.7.2. Non-gmail email address as Google Account
  • First create a Gmail account abc@gmail.com then go to My Account > Personal Info & privacy > Your personal info > Email > Advanced > Alternate emails > Add alternate email
  • This is to add an alternate non-gmail email address to your Gmail email account
  • These alternate non-gmail email addresses can be used to signin, recover password, get notifications and more
  • Using Google products e.g. Google Docs, will show your primary gmail email address
38.31.7.3. Bookmark to go directly to a Gmail account

https://mail.google.com/mail/u/?authuser=myemail@gmail.com

38.31.7.4. Filter and Search Operators
  • https://support.google.com/mail/answer/7190?hl=en
  • Search Operators are used inside the search box
    • from:(a@a.com) to:(b@b.com) subject:(Update Changes #*)
    • OR or {}
      • from:(a@a.com OR b@b.com)
      • from:a@a.com OR from:b@b.com
      • {from:a@a.com from:b@b.com}
      • from:(a@a.com OR b@b.com) ("4326737" OR "898-849-9531")
    • has:nouserlabels
    • has:userlabels
    • is:unread is:read
    • category:updates
    • - e.g. Not starred -is:starred
38.31.7.5. Shortcuts
  • Action
    Toggle star
    s
  • Navigation
    Go to threadlist, go up
    u

38.31.8. Google Sheets

38.31.8.1. Shortcuts
Menu
Help > Keyboard Shortcuts or C-/
  • Enable compatible shortcuts
  • Search a shortcut
(no term)
Insert row/columns or Open insert menu
  • C-M-= or C-M-S-=
Insert current date time
⇧ ⌘ ;

38.31.9. Google Maps

Embed single location
https://www.embedgooglemap.net/
(no term)
Embed multiple locations, use My Maps of Google Maps

38.31.10. Google Correlate

Use it to find other search keywords that are related to your keyword!

38.31.11. Google Planning Tools

https://www.thinkwithgoogle.com/tools/

38.31.12. Tools for Web Developers

38.31.12.1. Lighthouse in DevTools chrome:lighthouse

Refer to npm:lighthouse DevTools > Audits tab (among Elements, Console, etc.)

View and compare json report files
https://googlechrome.github.io/lighthouse/viewer/
Save as Gist
save report to a secret Gist under your GitHub account. The viewer link with Gist is publicly available
38.31.12.4. Fetch as Google google:search console
For sites that you don't own
https://technicalseo.com/seo-tools/fetch-render/ or do an iframe like this and then use Fetch as Google
<html>
  <head><title>Fetch &amp; Render Proxy</title>
  </head>
  <body style="margin:0px;padding:0px">
    <iframe src="http://www.example.com/" frameborder="0" style="overflow:hidden;height:10000;width:1024" height="10000" width="1024"></iframe>
</script>
</body>
</html>
38.31.12.5. Rich Snippet Testing Tool, Structured Data Testing Tool google:json-ld

Test websites that you own and don't

38.31.12.6. Google Site Status (security) google:site status

Google Site Status :: https://transparencyreport.google.com/safe-browsing/search

38.31.12.7. Install with headless chromium
curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
sudo apt install -y nodejs

# install chromium browser
sudo apt install chromium-browser
# or
# sudo apt-get install chromium

npm i -g lighthouse

lighthouse --chrome-flags="--headless" https://github.com

38.32. Translation

tool:Poedit Open .pot and .po files in Poedit

Online Editor. Send .po file to customer who will translate online and they can save and send back the translated .po!

38.33. HTML Minimizer, document.write

Online minimizer

Node.js minimizer

tool:html:document.write Online convert a single line html to document.write

document.write("string") where string has to escape:

  • double quotes \"
  • backward slash \\
  • forward slash \/ as "</tag>" causes an error

Use document.write is not a good idea and the usage will be blocked only when all the following conditions are met:

  • The user is experiencing a very poor network connectivity,
  • The script is parser-blocking (neither async nor defer attributes) and is not already in the browser cache,
  • The instruction is added in the top level document (e.g. iframes won’t be concerned),

Use e.innerHTML('') to insert elements, refer to js:external script to load javascript

38.35. File Transfer

https://send.firefox.com
up to 2.5GB
(no term)
WeTransfer
Free
file size send limit 2GB
Pro
$12/month
  • 1 TB
  • 20 GB one file
  • Your own branding page with custom domain and URL

38.36. CloudConvert.com

38.37. Online App Publishing

38.37.1. https://glitch.com

  • Node.js with SQLite, Webpage
  • Projects created while not signed in will be deleted after 5 days
  • Change project name e.g. abc-xyz
    Public access
    https://glitch.com/~abc-xyz
    Edit project file at public/index.html
    https://glitch.com/edit/#!/abc-xyz?path=public/index.html:1:0
  • anonymous viewers can't see it. When remixing, it's cleared and it's not copied
  • .data folder to store data. When remixing, it isn't copied
  • Refer to vs:code:ext:glitch.glitch
  • Project Setting
    • Turn off Refresh on app change otherwise it always refresh for the root project url
  • run refresh in console

38.37.2. https://www.bitballoon.com

Webpage

38.37.3. https://zeit.co/

Node.js, Dockerfile

38.38. Email Service and Email Marketing

38.38.1. SendGrid

  • Create a user per website who can use to authenticate SMTP and API
  • Sign to main account and go to Settings > Credentials
  • Add a header X-SMTPAPI as JSON to assign category shown on SendGrid.com Stats and Activity
    • Drupal way $message['header']['X-SMTPAPI'] = '{"category": ["fromwebsiteA"]}'
  • Drupal
    • Use the SMTP Authentication Support module (smtp) and go to /admin/config/system/smtp to config
    • Install Options - turn this module on
    • SMTP server: smtp.sendgrid.net
    • SMTP port: 465 (587 is being used…)
    • Use encripted protocol: Use SSL
  • Wordpress
    • Install plugin SendGrid Refer to GitHub for setting config in wp-config.php
38.38.1.1. Whitelabel
  • Domain whitelabel
    • adds a SPF (Sender Policy Framework) record (dns:spf) to your domain and DKIM (DomainKeys Indentified Mail) entries, which authorizes and authenticates SendGrid sending on behalf of your domain. It basically removes "sent on behalf of SendGrid" or "via" message that some email providers dispaly
    • The sent domain is the domain of the email address from "From"
    • Usually, you need to create one subdomain per main domain. e.g. delivery.yourdomain.com
    • With Automated Security checked, SendGrid will show you to create 3 CNAME records for your DNS registrar
    • Some DNS registrar might not support _ in subdomains in CNAME host. In this case, don't check Automated Security
    • Without Automated Security checked, 1 MX and 2 TXT records are provided from SendGrid
  • Email link whitelabel
    • adds a CNAME record for a a subdomain that you choose, which masks click and open-tracking links to your domain rather than a SendGrid domain. e.g. In email body there's a link yourdomain.com. Before email link whitelabel is setup, the sent out email with this link changed to SendGrid domain. After whitelabel is setup, this link is changed to e.g. click.yourdomain.com
    • This subdomain has to be different from the Domain whitelable subdomain
    • SendGrid will show you 2 CNAMES per subdomain setup
38.38.1.2. PHPMailer
date_default_timezone_set('America/Toronto');
require_once(__DIR__.'/../../../wp-includes/class-phpmailer.php');

$receipients = array('a@b.com');
$subject = '';
$body = '';
$html = <<<HTML
    <html>
    <head>
  <title>{$subject}</title>
    </head>
    <body>
  {$body}
    </body>
    </html>
HTML;

$mail = new \PHPMailer();
$mail->isSMTP();
//$mail->SMTPDebug = 2; // 0 for off in production
$mail->SMTPDebug = 0;
$mail->Host = 'smtp.sendgrid.net';
$mail->SMTPAuth=true;
$mail->Username= 'apikey'; // literally 'apikey'
$mail->Password= 'secret key';
$mail->Port = 587;
$mail->CharSet= 'UTF-8';
$mail->SetFrom('donotreply@myweb.com', 'My Website');

$mail->addCustomHeader('X-SMTPAPI', '{"category": ["fromwebsiteA"]}');

foreach ($reciepients as $v) {
  $mail->AddAddress($v);
}

$mail->Subject = $subject;
$mail->MsgHTML($html);

if (!$mail->Send()) {
  return 'error: '.$mail->ErrorInfo;
}
else {
  return 'success';
}
38.38.1.3. cURL sends emails
# basic
curl --request POST \
  --url https://api.sendgrid.com/v3/mail/send \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{"personalizations": [{"to": [{"email": "recipient@example.com"}]}],"from": {"email": "sendeexampexample@example.com"},"subject": "Hello, World!","content": [{"type": "text/plain", "value": "Heya!"}]}'

# send to multiple receipients
curl --request POST \
  --url https://api.sendgrid.com/v3/mail/send \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{"personalizations": [{"to": [{"email": "recipient@example.com"}],"cc": [{"email":"recipient2@example.com"}, {"email": "recipient3@example.com"}, {"email":"recipient4@example.com"}]}], "from": {"email": "sendeexampexample@example.com"},"subject":"Hello, World!", "content": [{"type": "text/plain", "value": "Heya!"}]}'

# scheduled
curl --request POST \
  --url https://api.sendgrid.com/v3/mail/send \
  --header 'authorization: Bearer YOUR_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{"personalizations": [{"to": [{"email": "recipient@example.com"}]}],"from": {"email": "sendeexampexample@example.com"},"subject":"Hello, World!","content": [{"type": "text/plain","value": "Heya!"}], "send_at" : UNIX_TIMESTAMP_HERE}'

38.38.2. Dialog Insight

38.38.3. DMARC, SPF, DKIM dns:dmarc

DMARC
Domain-based Message Authentication, Reporting, and Conformance
(no term)
DMARC serves as a method of authentication for your brand, alongside Sender Policy Framework (SPF) and DomainKeys Identified Mail (DKIM)
(no term)
What sets DMARC apart from the other two authentication methods is its reporting capability
(no term)
DMARC is an email authentication technology that protects a domain from being used in phishing and spoofing attempts by using a signing policy to define how receiving inbox providers should handle messages that fail an authentication check. DMARC also allows for a reporting mechanism in which inbox providers can send reports on email that appears to be sent from a certain domain back to the domain owner
(no term)
Inbox providers that support DMARC will attempt to validate both DKIM and SPF, and depending on the outcome of those checks, will look to the sending domain's DMARC policy on how to handle emails that fail authentication
(no term)
If the inbox provider is able to successfully validate either DKIM or SPF, the email will continue on its normal path. If the message fails both DKIM and SPF authentication checks, however, the inbox provider will enforce the sender’s DMARC policy, which specifies how the email should be handled if it fails authentication and where to send any reports
(no term)
DMARC has three levels of policy
None
If the policy is set to none, the receiving inbox provider monitors and reports on metrics
Quarantine
If the DMARC policy is set to quarantine, the inbox provider places messages that fail the required checks into the user's spam folder
Reject
If the policy is set to reject, the inbox provider will block any message that fails authentication
(no term)
https://sendgrid.com/blog/dmarc-domain-based-message-authentication-reporting-conformance/
DMARC Specifications on DNS Tags and Values
https://dmarc.org/resources/specification/
TXT _dmarc v=DMARC1; p=none; rua=mailto:dmarc@myweb.ca; ruf=mailto:dmarc@myweb.ca; fo=1; adkim=r; aspf=r;

Sample

v
must have value DMARC1 and should be the first tag
p (required)
Policy applies to the domain queried and to subdomains, unless subdomain policy is explicitly described using the sp tag. Possible values.
none
Domain Owner requests no specific action be taken
quarantine
Domain Owner wishes to have email that fails the DMARC check be treated by Mail Receivers as suspicious. This could mean "place into spam folder", "scrutinize with addtional intensity", and/or "flag as suspicious"
reject
Domain Owner wishes for Mail Receivers to reject email that fails the DMARC check. Rejection should occur during the SMTP transaction
adkim
default r. Indicates whether strict or relaxed DKIM Identifier Alignment mode is required by the Domain Owner. s for strict mode
aspf
same as adkim but for SPF Identifier Alignment mode
fo
default 0. Failure reporting options for generation of failure reports. This tag is ignored if ruf tag is not specified. Colon-separated of the below values
0
Generate a DMARC failure report if all underlying auth mechanisms fail to product an aligned "pass" result
1
Generate a DMARC report if any underlying
d
Generate a DKIM report if the message had a signautre that failed evaluation, regarless of its alignment
s
Generate an SPF report if the message failed SPF evaluation, regardless of its alignment
rua
Addresses to which aggregate feedback is to be sent. Comma-separated
ruf
Addresses to which message-specific failure info is to be reported. Comma-separated
pct
default 100. Percentage of messages from the Domain Owner's mail stream to which the DMARC policy is to be applied
sp
Requested Mail Receiver policy for all subdomains. Indicates the policy to be enacted by the Receiver at the request of the Domain Owner. It applies only to subdomains of the domain queried and not to the domain itself. If absent, the policy specified by the p tag must be applied
(no term)
Refer to dns:dkim and dns:spf
(no term)
For adding another subdomain as in From: e.mydomain.com
Add a SPF record that refers to the root domain's SPF record
TXT e v=spf1 include:mydomain.com -all
Add a DKIM record (may just duplicate the root domain's DKIM record)
TXT customer._domainkey.e k=rsa; p=xxx or CNAME to point to mail server and mail server's TXT record will have the final DKIM value. Refer to dns:dkim
(no term)
Ensure email has header DKIM-Signature that uses d=e.mydomain.com to match the From: e.mydomain.com

38.38.4. Exchange anti-spam message headers

  • https://docs.microsoft.com/en-us/office365/securitycompliance/anti-spam-message-headers
  • https://mha.azurewebsites.net/pages/mha.html
  • Refer to ms:scc
  • Exchange Online Protection (EOP) inserts to each email
    CIP:[IP Address]
    The connection IP
    IPV:NLI
    IP was not listed on any IP reputation list
    SFV:SFE
    Filtering was skipped and the message was let through because it was sent from an address on an individual's safe sender list
    SFV:SPM
    Indicates that the message was marked as spam because of the EOP spam filters
    SFV:BLK
    Indicates that the message was marked as spam because the sending address is on the recipient's Blocked Senders List
    SFV:SKS
    Indicates that the message was marked as spam prior to the content filter. This could include a mail flow rule (also known as a transport rule) marking the message as spam. Run a message trace to see if a mail flow rule triggered which may have set a high spam confidence level (SCL)
    SFV:SKB
    Indicates that the message was marked as spam because it matched a block list in the spam filter policy
    SFV:BULK
    Indicates that the Bulk Complaint Level (BCL) value located in the x-microsoft-antispam header is above the Bulk threshold that has been set for the content filter. Bulk email is email which users may have signed up for, but may still be undesirable. In the message header find the BCL (Bulk Confidence Level) property in the X-Microsoft-Antispam header. If the BCL value is less than the threshold set in the Spam Filter, you may want to adjust the threshold to instead mark these types of bulk messages as spam. Different users have different tolerances and preferences for how bulk email is handled. You can create different policies or rules for different user preferences
    SCL:[value]
    Spam Confidence Level
    -1
    non-spam coming from a safe sender, safe recipient or safe listed IP address (trusted partner). Deliver to inbox folder
    0, 1
    non-spam because the message was scanned and determined to be clean. Deliver to inbox folder
    5, 6
    Spam. Deliver to Junk Email folder
    7, 8, 9
    High confidence spam. Deliver to Junk Email folder
  • provides addtional info about bulk mail and phishing
    BCL
    Bulk Complaint Level
    0
    message isn't from a bulk sender
    1, 2, 3
    message is from a bulk sender that generates few complaints
    4, 5, 6, 7
    message is from a bulk sender that generates a mixed number of complaints
    8, 9
    message is from a bulk sender that generates a high number of complaints
    PCL
    Phishing Confidence Level
    0-3
    message content isn't likely to be phishing
    4-8
    message content is likely to be phishing
    -9990
    message content is likely to be phishing. Exchange Online Protection only
  • EOP inserts email authentication results

You can use

  • Exchange Admin Centre > Protection > Spam filter to create a block or allow list based on sender email address or sender domain
  • Exchange Admin Centre > Protection > Connection filter to create a block or allow list based on IP
  • only use it when filter criteria is complex e.g checking message headers or the names of attachments or add complex actions e.g. adding a disclaimer to the message or applying a time period where the rule is active

38.38.5. Best Practice, Spam Score

Google

Other Spam Score Tools

38.38.6. Preview text

Preview text is the hidden text that is the first element in the body and it should be at least 90 characters long. Img alt attribute values. 

<div style="display:none;font-size:1px;color:#333333;line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;">
  Insert preview text here.
</div>

<!-- Insert &zwnj;&nbsp; hack after hidden preview text. 10 characters -->
<div style="display: none; max-height: 0px; overflow: hidden;">
&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;
</div>

38.38.7. Klaviyo

  • Email marketing platform used by big companies
  • kmail-lists.com

38.38.8. Send test HTML email for free

https://htmlmail.pro/ use your Gmail to send HTML emails.

38.39. Payment Gateway

38.39.1. Chase E-xact Gateway

Chase Paymentech

https://support.e-xact.com/hc/en-us/articles/360000632374-Hosted-Checkout-Integration-Manual

Chase Payment Gateway

Payment Page Settings General Return to Your Site URL is hard coded. It's shown when a user comes to Payment Page and sees a Return to Website if the user doesn't want to proceed. And it also serves as a go back link when timeout happens. Receipt Page AUTO-POST, AUTO-GET don't display a receipt on Chase Payment Gateway but instead post the transcaction result to your website POST, GET display a receipt on chase Payment Gateway and a button including a link to GET or POST back to your website. Unfortunately, this link cannot be dynamic.

Relay Response Transaction result is POST to your site and your site responds with HTML which E-xact later displays. The response is usually a custom receipt. If response is not received in 25 seconds, E-xact will display its own receipt. You can give a dynamic relay response url in the submit form rather than a static url setup in E-xact Payment Page setting page. The form field is x_relay_url If replay response is required, in submit form you should add a hidden form field <input name="x_relay_response" value="TRUE" type="hidden"> Silent Post Same as replay response but the response is ignored. The sequence of silent post and relay response is not guaranteed!

Chase Payment Gateway Form Create a Payment Page and you will get transaction_key, page_id, response_key

Do not expose transaction_key, page_id, response_key!!

Form field for submit (required fields) x_login :: page_id x_invoice_num :: your internal order id $order_number x_fp_sequence :: Random number e.g. your internal order id. $order_number x_fp_timestamp :: Requests expire after 15 minutes. Time in seconds. php :: time(). $x_fp_timestamp x_amount :: positive number. $x_amount x_fp_hash :: string. HMAC-MD5 hash from the transaction_key and concatenation of the values for x_login, x_fp_sequence, x_fp_timestamp, x_amount and (if given) x_currency_code. Separated by ^. php : hash_hmac('MD5', $page_id. '^' . $order_number. '^' . $x_fp_timestamp. '^' . $x_amount. '^CAD' , $transaction_key); x_show_form :: 'PAYMENT_FORM', constant

Post back data and fields https://hostedcheckout.zendesk.com/hc/en-us/articles/114094066114#10.1 x_response_code 1 for approved, 2 for process but not approved, 3 for not processed including cancel

x_response_reason_code 1 for approved, 2 for declined, 3 for error x_response_reason_text

x_invoice_num :: the same as x_fp_sequence in TandT case x_fp_sequence

x_test_request boolean. If it's in test mode, true exact_ctr string. Receipt string. Successful or failed, always return a string. Always show! exact_issname string. Issuing Bank name exact_issconf string. Issuing Bank Confirmation Number x_fp_timestamp x_MD5_Hash

You need to validate the post back value if $_POST['x_MD5_Hash'] == md5($response_key. $_POST['x_login'] . $_POST['x_trans_id'] . $_POST['x_amount']);

38.39.2. Moneris Payment Gateway

https://developer.moneris.com/More/Testing/Testing%20a%20Solution https://developer.moneris.com/More/Testing/Penny%20Value%20Simulator

Visa
4242424242424242 4502285070000007
MasterCard
5454545454545454

38.39.3. Stripe

  • https://woocommerce.com/products/stripe/
  • as long as your yearly charges are below $1 million
    Credit and debit cards
    pay 2.9% + 30 cents per successful transaction
    (no term)
    https://stripe.com/en-ca/pricing
  • No China
  • Refund, recurring payments
  • Credit cards, debit cards, Apple Pay, Google Pay, Alipay and Payment Request API support
  • Team
    • An admin (one email address) can create multiple Stripe accounts
    • Admin can add members to manage a Stripe account
  • Products

    • Types

      Goods
      use Orders API
      Services
      subscriptions. Has following parameters
      • name
      • type
      • metadata
  • Plans
    • Has following parameters
      • product: e.g. ID of an existing product to associate with the plan
      • amount to charge a customer per subscription per interval
      • day, week, month and year. Use interval_count to set e.g. 30 days
      • unique ID, auto generated by Stripe
      • can't change amount, currency and interval after a plan is created. Can change metadata and nickname

38.39.4. PayPal Powered by Braintree on WooCommerce

PayPal Powered by Braintree
https://woocommerce.com/products/woocommerce-gateway-paypal-powered-by-braintree/
  • 2.9%+30 cents per successful credit card or digital wallet transaction (e.g. Apple Pay)
  • Credit cards and debit cards
  • Payment from buyers' PayPal accounts

38.40. Market Research market research:firm

38.41. W3C

  • Working Draft
  • Last Call
  • Candidate Recommenation (CR, Standard)
  • Proposed Recommendation (PR)
  • Recommendation (REC)
  • Working Group Note, Interest Group Note (NOTE) which may make Editor's Drafts (ED, API's may take this as a strong signal..)

38.42. Adobe

38.42.1. Acrobat Pro DC

38.42.1.1. Open the same file in separate windows

Window -> New Window

38.42.2. Creative Cloud

If you have error 146 and an app update failed, you might need to reinstall Creative Cloud Desktop App and then update individual app.

38.42.3. Photoshop

38.42.3.1. Image Resize
  • Preserve Details (enlargement)
  • Bicubic Smoother (enlargement)
  • Bicubic Sharper (reduction)

Resize an animated gif Window > Workspace > Motion

Crop if you like.

File > Export > Save for Web (Legacy) > Resize it there

38.42.3.2. Make a lighter or darker color
Shade
mix with black

  • RGB: Multiply each component by 1/4, 1/2, 3/4 etc. of its previous value. The smaller the factor, the darker the shade 

    $newR = $currentR * (1 - $shade_factor);
$newG = $currentG * (1 - $shade_factor);
$newB = $currentB * (1 - $shade_factor);
  • lower the Value / Brightness or increase the Saturation
  • lower the Lightness
Tint
mix with white

  • RGB: Calculate (255 - previous value) for each component, multiply that by 1/4, 1/2, 3/4, etc. and add that to the previous value. The greater the factor, the lighter the tint 

    $newR = $currentR + (255 - $currentR) * $tint_factor;
$newG = $currentG + (255 - $currentG) * $tint_factor;
$newB = $currentB + (255 - $currentB) * $tint_factor;
  • increase Value / Brightness or lower the Saturation
  • increase the Lightness
Tone

mix with with gray 

$newR = $currentR + ($aR - $currentR) * $alpha;
$newG = $currentG + ($aG - $currentG) * $alpha;
$newB = $currentB + ($aB - $currentB) * $alpha;

// Where
// black = (aR, aG, aB) = (0, 0, 0)
// white = (aR, aG, aB) = (255, 255, 255)
// greay = (128, 128, 128)
(no term)
PHP Gist
(no term)
To easier and more granularly manipulate shade and tint, convert Hex/RGB to HSL, adjust the lightness and then convert back to Hex/RGB

38.42.4. Illustrator

38.42.4.1. Setting

Preferences > General > enable Scale Strokes & Effects

Control Panel > Document Setup When nothing is selected, on the top below menu, you will see a Control Panel. Document Setup can set Units and Bleed.

View > Smart Guides Show Smart Guides (alignment suggestion) View > Show Grid Show grids inside and outside artboards

38.42.4.2. Panels, Workspace, Artboard

Add extra panels Window Menu

  • Align along with Transform and Pathfinder
  • Info without Navigator
  • Type > Character panel

Default Panels

  1. Layers Panel
    • Change path color
    • Template and Dim images to
      • That layer has raster images. Used as a background to draw vector path on top of it
    • Move objects inside or to a different layer
  2. Artboards Panel, Artboard Tool Shift + O
    • Change canvas size, orientation and other settings which you see when creating a document
    • May also change the current artboard in the Artboard Tool
    • Export individual artboard to different file format File > Export > Export for Screens
    • Show Center Mark (center point)

    Save as a new workspace. Reset to default workspace.

38.42.4.3. Hand, Zoom, Screen mode, View, Rulers, Guides

Hand Tool H or Spacebar to temporary change to hand tool

Zoom tool (z)

  • Alt + Click zoom out
  • Ctrl + 0 view whole or double click on hand tool
  • Ctrl + 1 view at 100% or double click on zoom tool

Cycle through screen modes for presentation F

A View is a view which shows a portion with correct zoom and boundaries.

View Menu > Create a new view

Show Rulers Ctrl + R

  • Drag from one ruler to create a guide line
  • Or double click on the ruler to create a guide
  • Lock Guides in View Menu > Guides > Lock Guides
  • View > Guides > Make Guides
38.42.4.4. Repeat the previous action

Ctrl + D e.g.

  • Duplicate an object after Alt + Drag
  • after scaling with the Copy action.
38.42.4.5. Move, Select

Shift + Arrow :: move 10px

Ctrol + click on an object :: temporary select an object Ctrol + Drag :: temporary select multiple objects

38.42.4.6. Pencil and Brush tools

Free hand draw. Pencil Tool (N) can only apply regular stroke. Double click on the Pencil Tool on the left to changeL

  • smoothness
  • Closing path pixel amount

When about to close the path, hold down Alt. When you see a circle in the cursor, release the mouse. The path will be closed.

Redraw part of the path to reshape part of the path

Brush (B) is the same as pencil but can add pattern and image in path.

  • Window > Brush
38.42.4.7. Basic Shapes
  1. Line, arc, spiral, rectangular grids, polar grid

    In the left menu

    • line (\)
    • arc
    • spiral (Shift, Up and Down to increase # of rings )
    • rectangular grids
    • polar grid
  2. Rectangle, Ellipse

    In the left menu

    • Rectangle tool (M)
    • Rounded Rectangle Tool
    • Up or Down to change the number of sides
    • Up or Down to change the number of angles.
    • Flare Tool

    For any shape tool

    • Alt + Click to draw a new shape with the same center point of another shape
    • Ctrl + Drag to make the shape fatter or skinnier.
38.42.4.8. Drawing Modes

Shift + D :: change drawing mode: normal, behind, inside

38.42.4.9. Transform Object
  1. Change Anchor Points or Control Handles

    Direct Selection Tool A

    • Alt + Drag on anchor point to change only on one side
  2. Group, Ungroup Objects, Isolation mode, Lasso Tool
    • Ctrol + G Group multiple objects
    • Shift + Ctrol + G Ungroup

    Isolation mode: Double click on a group to go to sub group

    Lasso Tool Q

    • Manual select multiple objects and then group them.
  3. Duplicate Object

    Alt + Drag :: with Shift to duplicate it horizontally or vertically Repeat the previous action Ctrl + D

  4. Offset an object

    Offset anchor points to create another object Object > Offset Path

  5. Scale, Shear

    Scale Tool S. Double click to scale numerically. May also change the reference point to scale to withtout Double click.

    Shear Tool. Double click to shear numerically.

  6. Rotate and Reflect Object

    Rotation Tool R Choose another reference point to rotate Alt + Drag while rotating and release will make a copy after rotation Then may repeat the action to make several copies Ctrl + D

    Reflect Tool O Symmetry When the tool is selected, the reference point by default is the center point. Change the reference point and then drag. Use Shift to rotate vertically or horizontally Hold down Alt while drag and then release to copy the reflection object

    May also right click on the object, select Transform > Reflect and select copy. Later move that object in place

  7. Free Transform Tool

    Mouse operations only E Rotate, scale, change aspect ratio

  8. Transform Each

    Object Menu > Transform > Transform Each Ctrl + Alt + Shift + D

    Transform based on one of the 9 points.

38.42.4.10. Fill and Stroke

Default Fill and Stroke D No fill \ Select multiple objects, press I, then select a color. All objects will be filled in that color.

  1. Gradient

    Select an object, go to Window > Gradient, select a Type.

    On the left menu, select Gradient Tool G, then draw line inside the object.

    May apply gradients to both stroke and fill for an object.

  2. Stroke

    Show outline only Ctrl+Y Click on Stroke on the top menu (Control Panel), when a path/object is selected.

    May change the width of a stroke partially or dynamically using Width Tool Shift + W

38.42.4.11. Color, Swatch
  1. Color Setting

    Edit > Color Settings North America General Purpose 2 RGB: sRGB IEC61966-2.1 CMYK: US Web Coated SWOP v2 Choose Preserve for RGB and CMYK Enable all Ask When Opening, Ask When Pasting

  2. Process, Global, Spot Color

    Process color means that color only applies to an object. Changing that process color only changes color of that object. Changing a global color means all objects using that global color will change color.

    To create a global color Open Window > Swatches Apply a color for an object. Select or create a group. New Swatch, select Global

    Spot color is usually provided by a vendor/company which, for example, provides specific ink. To further ensure the spot color is always accurate on finished products (e.g. prints).

  3. Import Export Swatch

    .ase file. Window > Swatches

38.42.4.12. Appearance

Window > Appearance Add overall effects on all objects or on a specific object Change attributes: Stroke, Fill, Opacity May stack stroke and fill effects. May use pattern in Fill Add Live Effects (fx) or Effect (top menu): Drop Shadow, Warp etc. Save appearances as graphic styles Select an object with some Appearances, then open Window > Graphic Styles, create new Apply that new Graphic Style on another object in the same .ai file. In the Graphic Styles window, Select unused graphic styles, and remove. Save the rest Graphic Styles, and hit Save Graphic Styles as Library. Open a new ai file, in the Graphic Styles Window, load User Defined. Then you can apply the Graphic Style across ai files.

38.42.4.13. Complex Shapes, Compound Path, Pathfinder, Shape Builder

Compound Path

  • 2 Paths: Inner and outer circles
  • Select both paths and Ctrl+8 to make a compound path
  • Later double click on the compound path, inner circle can be resized!

Pathfinder

  • Window > Pathfinder. You can't change the paths after it's applied
  • Shape modes: the result is a single path of the end shape
    • Unite
    • Back path mius front path. Punch a front path hole in back path.
    • Intersect
    • Front + Back - intersect
  • Pathfinders mode: the result is a bunch of paths that form the end shape
    Divide
    Front + Back but with shapes cut at any intersection
    (no term)
    Trim
    (no term)
    Merge
    (no term)
    Crop
    (no term)
    Outline
    (no term)
    Minus Back

Shape Builder

  • Left menu: Shift + M
  • Result is like Pathfinder's Pathfinders mode.
  • Select both paths and draw straight line to indicate which area should bring front
  • Draw straight line while holding Alt is to subtract
38.42.4.14. Eraser

Eraser fixes path around edges. Shift + E [ or ] to increase the size

Make sure to select a path then erase. Otherwise all paths will be erased.

38.42.4.15. Pen Tool

Pen Tool P

  • Point, click and move to create straight line
  • Alt + Drag to bend any path at any point
  • Draw curve line with Pen Tool
    • Start point: Click and drag to the direction that the curve starts
    • End point: click and drag to the opposite direction that the curve ends
    • Change handle pdirection of an anchor point
      • Direct Selection Tool A and change the handle direction
    • Remove "whip": remove the handle of an anchor point
      • Alt + Click on the anchor point
  • No fill or fill none \ when necessary
  • Direct Select Tool A to change anchor points and handles

Add Anchor Point Tool + Delete Anchor Point Tool - Anchor Point Tool Shift + C

  • Turn a point into anchor point (no handle, angle)
  • Click + Drag to create 2 handles
38.42.4.16. Type Tool

T Convert between Point Type and Area Type Type > Convert to Point Type

Point Type
one line. It scales when it's resized
Area Type
Fit text into a box. It doesn't scale when it's resized

Panels

  • Window > Character
    • Character, Paragraph, OpenType
  • Window > Character Styles
    • Used to save Character or Paragraph styles defined above
    • Character Styles, Paragraph Styles

Wrap text around an object

  • Select the object that text should wrap around
  • Object > Text Wrap > Make
  • Object > Text Wrap > Text Wrap Options to change the offset
  • Right click on the same object, Arrange > Bring to Front

Type on a Path Tool

  • e.g. curve path

Change Font Size Shortcuts

  • shift + ctrl + > or < 2 points at a time
  • shift + alt + ctrl + > or < 10 points at a time

Change Tracking of Font (horizontal space between characters)

  • opt + right or left arrow

Paragraph spacing opt + up or down arrow

Adobe Typekit :: Sync Fonts with local Desktop as long as Creative Clouds are installed locally

Turn Type into Paths Type > Create Outlines

38.42.4.17. Copy from InDesign
  • Select objects in InDesign and paste in AI.
  • View > Outline, you will clipping mask. Get rid of it!
  • Object > Clipping Mask > Release
  • Select the outest outline and delete it.
  • View > Preview to turn off Outline view.
38.42.4.18. Resize canvas to selection

Object > Artboards > Fit to Selected Art

38.42.4.19. Raster Image

Place an image File > Place The image is linked from Illustrator to the image file. Illustrator can see any changes on the image file.

Links Panel Window > Links

Embed and Unembed (relink)

Clipping Mask

  • Draw a circle shape on top of the image
  • Select both the circle and the image
  • Object > Clipping Mask > Make or right click Make Clipping Mask
  • Double click to change the circle or image
  • Later right click on the image, Release clipping mask to remove the circle mask

Image Trace Select the image, Window > Image Trace

Convert Image Tracing to Paths Select the Image Tracing object and Object > Expand or Expand in the top Image Tracing control panel Once expanded, image cannot be traced again.

38.42.4.20. Crop image

You CAN crop raster images in Illustrator.

Draw a rectangle above the raster image. Select both the rectangle and the image choose Object > Clipping Mask > Make. change blending mode to "darken" from transparency tab choose Object > Flatten Transparency..> OK Choose Object > Expand… Now you have trimmed raster object.

38.42.4.21. Export Assets

Window > Asset Export. Add objects one by one to Asset Export

38.42.4.22. Export SVG

Select the object on the artboard then File > Save As > svg Turn font into path :: Select all text > Type > Create Outlines

CS6

  • Save As all objects on one or multiple artboards.
  • Can't Export selection or Export multiple artboards
  • The artboard size is the viewport size in SVG

CC version

  • All or selected artboards can be Exported as SVG.
  • Selected objects can be exported separately using File > Export Selection
  • Add an object as Asset and Export Asset
  • Copy the object and paste in Dreamweaver!

SVG Profile :: SVG 1.1 Fonts.

  • Type: SVG. If there's small text, use Convert to Outline which converts all Type to paths
  • Subsetting: None
  • Embed. Usually don't include raster image in SVG
  • Don't check Preserve Illustrator Editing Capabilities because it increases file size

Styling :: Inline Style or Presentation Attributes Font :: Convert to Outlines Images :: Preserve Minify but not Responsive

38.42.4.23. Scripts

Save script file in C:\Program Files (x86)\Adobe\Adobe Illustrator CS6\Presets\en_US\Scripts\

Restart AI and the script can be executed under File > Scripts >

38.42.5. Color CC

https://color.adobe.com/create/color-wheel/ https://www.lynda.com/Creative-Cloud-tutorials/Kuler-Essential-Training/136175-2.html

Analogous Create variants of the Base color which is always the center point. All 5 colors have the same saturation but differ in hue.

Monochormatic Same Hue value, but different saturation and brightness values.

Triad Create contrast colors.

Complementary Opposite colors.

Compound 2 sets of opposite colors and for each set create another analogous color

Shades Only different in brightness.

Use Mobile App: Adobe Capture to capture colors.

38.43. Digital Edition

38.43.1. Issuu

38.43.2. Pugpig

  • Clients
    • GQ, Scientific America, The Economist, The Sun

38.44. Publication

38.44.1. Production

38.44.1.1. ppi Media
  • Clients
    • New York Times
38.44.1.2. AdFlow Systems GmbH
  • German
  • Clients
    • Montreal newspapers

38.44.2. Market research market research:service

  • Refer to market research:firm
  • Cision

    • Earned media management software and services

      • Distribute content: print, radio, television, web and social media

      • CEDROM-Sni

        • Digital reproduciton rights (buy and sell copyright)
        • Digital media monitoring and analytics
      • Target key influencer
      • Measure impacts
  • https://www.comscore.com/
    • 30M Canadian Digital Audience, 11M Tablet, 18M Smartphone, 28M Desktop - 2017
    • Computer usage peaks at 7-8AM and all devices usage peak at 5-8PM - 2017
    • Mobile app usage is 86% compared to Mobile Web is 16% - 2017
    • Total time spent for all internet across devices: Desktop:38%, Smartphone App:38%
    • Total time spent for social media acrss devices: Desktop:21%, Smartphone App:50%
  • Quantcast
    • AI driven, audience insight from all websites use Quantcast. e.g.
      • What sites a reader visited
      • Keywords searched
      • Purchase habits
    • The database is called Quantcast Intelligence Cloud (QIC) powered by Q engine

38.45. CRM

38.45.1. HubSpot

38.45.1.1. Understanding the CRM
38.45.1.2. Pricing
  • Categories
    • Marketing Hub
    • Sales Hub
    • Service Hub (Custom Service)
    • CMS Hub
    • Operations Hub
  • Level
    • Free Tools
    • Starter
    • Professional
    • Enterprise
38.45.1.3. Workflow (Webhooks)
Intro
https://developers.hubspot.com/blog/implementing-webhooks-in-hubspot
(no term)
Part of Operations Hub
  • Free is already powerful
Webhooks API
https://developers.hubspot.com/docs/api/webhooks
(no term)
Use UI to setup a webhook to trigger events
38.45.1.4. API
  1. Engagement Entity
    1. Notes

      https://community.hubspot.com/t5/APIs-Integrations/Attach-file-to-deal-record-using-API/m-p/719945#M58797

  2. Files
38.45.1.5. Different Accounts
  • https://developers.hubspot.com/docs/api/account-types
  • Standard
  • App developer accounts
  • Developer test accounts
    • Up to 10 developer test accounts can be created under a app developer account
    • It's an Enterprise level account! Expires every 90 days
      • To renew a developer test account, switch to the app developer account, click on Testing tab in between Apps and App Marketplace and select a developer test account to renew trial

38.46. Automations

38.46.1. Zapier

38.46.1.1. CLI
  1. Dockerfile: npm install -g zapier-platform-cli
    FROM node:16

# Place global NPM dependencies in a non-root user directory
ENV NPM_CONFIG_PREFIX=/home/node/.npm-global
# optionally if you want to run npm global bin without specifying path
ENV PATH=$PATH:/home/node/.npm-global/bin

# Install global NPM modules
RUN npm install --global zapier-platform-cli
COPY .zapierrc /root
  2. Example App
    Tutorial
    https://developer.zapier.com/cli-guide/clone-the-example-app
    git clone https://github.com/zapier/zapier-platform-example-app-github.git
cd zapier-platform-example-app-github
npm install
  3. Environment Variable project/.env file
    Refer to a variable
    process.env.MY_ENV_VAR_1
  4. GitHub zapier-platform
    1. More Example Apps
      To bring an example package custom-auth
      zapier init [your app name] --template custom-auth
  5. CLI Commands
    1. zapier login --sso
      # Use single sign-on
zapier login --sso
    2. zapier register
      • https://platform.zapier.com/reference/cli#register
      • Create a new integration app and link the local app (link only once) to that integration

        • .zapierapprc is created and it looks like this 

          {
  "id": 123456,
  "key": "App123456"
}
        • If .zapierapprc exists, it will ask if you want to update the currently-linked integration as opposed to creating a new one
        • After zapier register, zapier push can be run to build and upload the new integration for use in the Zapier editor.
    3. zapier push
      • Before push, better do zapier validate
      • After push, the follow files are created/updated under ./build
        • build.zip
        • source.zip
    4. zapier integrations
      • If you find the app and the version is not listed, you will need to re-login again zapier login --sso
    5. zapier logs
      # Recent HTTP traffic to my API
zapier logs --type=http

# More detailed
zapier logs --type=http --detailed

# In the app code, throw custom debug messages using `z.console.log("my debug message")` and catch it in the logs:
zapier logs --type=console

# More opptions for the zapier logs command
zapier help logs
  6. Field types
38.46.1.2. Rate Limits and Throttling

https://help.zapier.com/hc/en-us/articles/8496181445261

38.47. Form Service

  • JotForm
    • Video uploading
    • 38.39
    • Dropbox
    • Voting

38.48. Survey Service

SurveyMonkey
Survey Monkey

  • Pricing

    • Business Plan
    • Team Advantage $27 CAD/user/month
    • Team Premier $75 CAD/user/month
    • Seat Subscription Renewal (1 user/year, $780 from 2019 to 2020. Manual payment)
      • Personal Plan
  • Phone Support
    • If the plan has phone support, fill up this contact form with I need more help, then support will call you
  • Question Types
    • multiple choices, short answer, dropdowns and checkboxes
    • Sliders, payment-acceptance fields (via Stripe) and dropdown matrices
    • File upload
(no term)
SurveyGizmo
  • Plan: https://www.surveygizmo.com/plans-pricing/
    • Full Access - $199/month

  • Reports

    • TURF report: Total Unduplicated Reach and Frequency
    • 100 respondents, 90 like cake A,B,C, the other 10 like cake X and Y. Restaurant owner can choose 3 new cakes to make. Result: A (or B, C), X, Y
  • Integration
    • API access
    • Salesforce Marketing Cloud
    • Google Analytics
    • Google Spreadsheets
    • Webhooks

  • Sharing

    • Permalink
    • iframe,
    • Buy US respondents!
    • Download, install on mobile and desktop, collect responses offline
    • QR code
    • Website Intercept
    • Banner (top/bottom), modal dialog, inline intercept
    • JS code on every page
      • sg_beacon( 'init', 'xxxyyyzzz') inits and starts sending beacon
      • During init, check trigger. Trigger requires another init call or page refresh/new pageview
(no term)
wp:plugin:crowdsignal

38.49. Merchandise

38.49.1. Amazon Associates

  • Join
    • Have to have an Amazon account
    • Payee info
    • Top level domains of websites and/or mobile apps on which you plan to display banners, widgets, Special Links, or other ads from Amazon Associates

38.50. HR

  • Use Radar chart or spider chart to compare candidates
  • Criteria
    • Education
    • Knowledge, Skills and Abilities (KSA)
      • communication
    • Behavior
      • Creativity
      • Self-directed
      • Relationship builder
      • Tolerance of ambiguity
      • Flexibility
      • Persuasiveness
      • Attention to detail
      • Results orientation
    • Organizational Compatibility
      • Career path relevancy
      • Genuine interest
      • Values/mission alignment
      • Commitment to higher education
      • Civically engaged
    • Diversity
      • Multilingual
      • Nontraditional career path
      • International experience
      • Equal opportunity initiatives
      • Experience working with diverse communities
      • Experience working with people who require reasonable accommodation
      • Previous experience managing conflict across differences

38.51. Training

38.51.1. Microsoft Virtual Academy MVA

Free https://mva.microsoft.com Put in the Exam code, e.g. 70-480 to get free training resources

38.51.2. Microsoft Certification exam list

https://www.microsoft.com/en-us/learning/exam-list.aspx

38.51.4. Microsoft Online Proctored Exams

https://www.microsoft.com/en-us/learning/online-proctored-exams.aspx Install the software and do a hardware test. Sign in http://microsoft.com/learning to take an exam half an hour early Click "Your benefits & exams" then "Start a previously scheduled online proctored exam"

38.51.5. HackerRank

38.51.6. CoderByte

38.52. Work Ethics

38.52.1. Remove Credentials

  • Windows Credential Store Panel
  • Chrome
  • Git remove remotes git remote rm li this also deletes all branches of that remote
  • ~/.ssh/config
  • rm -rf /mnt/e/li/
  • phpStorm remove all projects

38.53. TeamViewer

1）完整卸载并删除现有TeamViewer安装过的版本 2）删除：%AppData%\Teamviewer、%tmp%\TeamViewer 3）删除：C:\Users\Administrator\AppData\Local\TeamViewer 4）删除：HKCU\Software\TeamViewer、HKLM\SOFTWARE\TeamViewer 5）下载Teamviewer对应版本，安装方式选个人非商业用途，然后打补丁即可！

如果想让破解补丁用于便携版，可装一次安装版，然后打完补丁后把下面三个文件： TeamViewer.exe、TeamViewer_Service.exe、TeamViewer_Desktop.exe 复制出来替换到便携版即可！

http://www.zdfans.com/6351.html

38.54. Font

WOFF
font:woff
  • supported in Edge and modern browsers. For web, always include WOFF (Web Open Font Format) > TTF > OTF
  • It's good for SEO
  • It has TrueType Hinting and OpenType Features
EOT
IE 11 and less
OTF or TTF
For local computers, install OTF (OpenType Font) over TTF (TrueType Font)
(no term)
Refer to css:font-face

38.54.1. Abbreviations Foundary

  • https://www.extensis.com/blog/fonts/abbreviations-font-names/
  • usually in the form of one or two letters at the beginning or end of the name (LT, MT, A, BT, FB, URW). “Foundries” are the companies that create fonts, a term going back to the days of metal type
  • comes at the end of a name (Cyr, Grk, CE). Generally this only applies to older fonts where a separate font was issued for different languages. In most cases, newer fonts put all the languages in a single font
  • (Text, Display, Poster/Caption, Small Text, Regular, Subhead, Display)

38.54.2. TS: Not a valid font file Windows

Turn off firewall or try font converter to convert the font and then install

38.54.3. Line height

Fonts rendered different line height across browsers is because Line Height Adjustment option is not in the font.

Line Height Adjustments

  • Best (Use the best method to normalize the line height for this font)
  • 120% (Redefine the line height as 120% of the point size)
  • Automatic (Distributes OS/2.Typo values across ascender, descender and line gap)
  • *Bounding Box (Match the bounding box of the glyphs, line gap will always be 0)
  • Native (Use the line height as defined in the font, results may differ between browsers)

Use Bounding Box.

https://iamvdo.me/en/blog/css-font-metrics-line-height-and-vertical-align

p { * font metrics * –font: Catamaran; –fm-capitalHeight: 0.68; –fm-descender: 0.54; –fm-ascender: 1.1; –fm-linegap: 0;

* desired font-size for capital height * –capital-height: 100;

* apply font-family * font-family: var(–font);

* compute font-size to get capital height equal desired font-size * –computedFontSize: (var(–capital-height) / var(–fm-capitalHeight)); font-size: calc(var(–computedFontSize) * 1px);

* compute line-height:normal and content-area's height * –lineheightNormal: (var(–fm-ascender) + var(–fm-descender) + var(–fm-linegap)); –contentArea: (var(–lineheightNormal) * var(–computedFontSize));

–distanceBottom: (var(–fm-descender)); –distanceTop: (var(–fm-ascender) - var(–fm-capitalHeight));

–valign: ((var(–distanceBottom) - var(–distanceTop)) * var(–computedFontSize));

–line-height: 3; line-height: calc(((var(–line-height) * var(–capital-height)) - var(–valign)) * 1px); }

38.54.4. Open source alternative, identify font

38.54.7. iOS Font List

38.54.8. Extensis

https://www.extensis.com/support Download Universal Type Client (6.1.5) not Universal Type Server 6 Core Client

38.54.9. Adobe Fonts former Adobe TypeKit (AF)

  • https://fonts.adobe.com/about
  • Pricing
    • Complete library of fonts is included with
      • Creative Cloud All Apps subscription
      • Creative Cloud any single-application subscription
      • Each seat of a Creative Cloud team subscription
      • the Photography Plan
    • Some or full font library is included in some educational licensing programs
  • Font Pack
    • Showcase a group of fonts, then you can select individual font to fav, add, activate
  • Activate a font to use on desktop apps with a subscription plan
  • Web project
    • No limits on pageviews and domains for any web projects
    • Character set or subset
      Default
      English, French, German, Italian, Portuguese and Spanish
      (no term)
      All Characters
      Dynamic Subsetting
      required for East Asian web font

38.54.10. Adobe Edge Web Fonts (AEWF)

38.54.11. Google Font (GF)

  • Can be downloaded and installed on Desktop as .ttf and for website .woff or .woff2
  • https://fonts.googleapis.com/css?family=Tangerine|Inconsolata|Droid+Sans
  • https://fonts.googleapis.com/css?family=Tangerine:bold,bolditalic|Inconsolata:italic|Droid+Sans
  • http://fonts.googleapis.com/css?family=Source+Sans+Pro:300,400,600,300italic,400italic,600italic

  • Subset / scripts, Latin subset is always included if available and need not be specified 

    <link rel="stylesheet"
  href="https://fonts.googleapis.com/css?family=Roboto+Mono&subset=greek">
  • https://fonts.googleapis.com/css?family=Inconsolata&text=Hello%20World
  • effect">https://fonts.googleapis.com/css?family=Rancho&effect=shadow-multiple
    font-effect- class prefix
    <div class="font-effect-shadow-multiple">This is a font effect!<div>

38.54.12. FontForge

  • http://fontforge.github.io/en-US/
  • Element > Font Info > General to get Em Size
  • Element > Font Info > OS/2 to get
    • Win Ascent, Win Descent for Windows
    • H Head Ascent, H Head Descent for Mac OS
    • Capital Height
    • X Height
    • Typo Line Gap and H Head Line Gap

38.54.13. Social Media Icons wp:plugin:simple-social-icons

38.54.14. Monospace

i1lI|!j oO0 e3 4AF f2 7TJyt __ ;: "'` Xx* [{( sS xX cC jJ uU vV zZ
abcdefghijklmnopqrstuvwxyz
ABCDEFGHIJKLMNOPQRSTUVWXYZ
0123456789
~`!@#$%^&*()-_"'
[{()]}\/><,.?
38.54.14.2. Hasklig

https://github.com/i-tu/Hasklig extends Source Code Pro

38.54.14.3. Source Code Pro

38.54.15. Material Font, GF

<link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet">

<!-- IE 10+ -->
<i class="material-icons">face</i>

<!-- For browsers do not support ligatures, -->
<i class="material-icons">&#xE87C;</i>

https://fonts.google.com/icons (https://material.io/icons/) and corresponding codepoint index

Recommended sizing and color setting 

/* Rules for sizing the icon. */
.material-icons.md-18 { font-size: 18px; }
.material-icons.md-24 { font-size: 24px; }
.material-icons.md-36 { font-size: 36px; }
.material-icons.md-48 { font-size: 48px; }

/* Rules for using icons as black on a light background. */
.material-icons.md-dark { color: rgba(0, 0, 0, 0.54); }
.material-icons.md-dark.md-inactive { color: rgba(0, 0, 0, 0.26); }

/* Rules for using icons as white on a dark background. */
.material-icons.md-light { color: rgba(255, 255, 255, 1); }
.material-icons.md-light.md-inactive { color: rgba(255, 255, 255, 0.3); }

/* Custom color*/
.material-icons.orange600 { color: #FB8C00; }

<i class="material-icons md-24">face</i>

<i class="material-icons md-dark">face</i>
<i class="material-icons md-dark md-inactive">face</i>
<i class="material-icons md-light">face</i>

<i class="material-icons orange600">face</i>

38.54.16. Poppin, GF

38.54.17. Museo, GF

38.54.18. Roboto Slab, GF

38.54.19. IcoMoon

Upload font to IcoMoon, and you can generate SVG sprite!

38.54.20. Fontello

Open source Font, CSS, SVG sprites. Fontelico Font Awesome Entypo Typicons Iconic Modern Pictograms Meteocons MFG Labs Maki Zocial Brandico Elusive Linecons Web Symbols

38.55. Digital Agency

38.55.1. PodBean

38.55.2. Castos.com

38.55.3. Exercism.io

39. Industry Compliance, Standards

39.1. WCAG, AODA, Web Content Accessibility Guidelines

Checking Tools

https://www.w3.org/WAI/WCAG20/quickref/

Keyboard accessbile for non link elements and non form elements 

<div onclick="doSumat();" onkeypress="doSumat();" tabindex="">

39.1.1. <a>

Multiple read more on a page 

<a href="babymayor.html" aria-label="Read more about Seminole's new baby mayor">[Read more...]</a>

39.1.2. iframe

<iframe title="" style="border:0px;">

frameborder="0" is obsolete.

39.1.3. input, label

if <input> has no <label>, then add title attribute

Remove size attribute for input type file, confirm_email 

$html = preg_replace('/(<input type="file" [^>]+) size=".*?"/si', '$1', $html);

39.1.4. pdf

  • PDF and Office documents need to set Title, Advanced > Language in Document Properties
  • PDF and Office document image has to have title attribute.
  • Edit > Manage Tools > add Accessibility to the right pane
  • Accessibility > Full Check > Click on items need alternate text to find on page and then doubleclick to select and then right click Tag as Figure
    • Accessibility > Autotag Document will retag the whole document. The best approach is to add tag to image/figure because autotag might add links wrong based on text context.
    • Do these
  • PDF/UA-1 compliance. PDF needs to have this XMP metadata to mark it as PDF/UA-1 compliant
    • Tools > Print Production > Preflight, choose PDF Standards or Acrobat Pro DC 2015 Profiles as profile, choose the wrench tool icon (Select single fixups), expand to Document info and Metadata and choose Set PDF/UA-1 entry and then Fix

39.1.6. Skip to content link

Right below <body> 

<a id="skip-nav" href="#main-content">Skip to content</a>
<main id="main-content"></main>

#skip-nav {
  position: absolute;
  top: -1000px;
  left: -1000px;
  height: 1px;
  width: 1px;
  text-align: left;
  overflow: hidden;
}
#skip-nav:active,
#skip-nav:focus,
#skip-nav:hover {
  width: auto;
  height: auto;
  position:static;
  overflow: visible;
}

39.1.7. Color Contrast Ratio

WCAG 2.0 level AA requires a contrast ratio of 4.5:1 for normal text and 3:1 for large text. Level AAA requires a contrast ratio of 7:1 for normal text and 4.5:1 for large text. https://webaim.org/resources/contrastchecker/ http://leaverou.github.io/contrast-ratio/

39.2. SR&ED: Scientific Research and Engineering Development

  • Format
    • Technical Uncertainties
      • the main objective of this project was to …
    • Work Done
      • we hypothesized that it would be possible to do … in order to achieve the main ojective
      • the project was completed? Y/N
    • Technological Advancements
      • we aimed to achieve the following: …
  • Besides the format, we need to explain:
    • Why is the problem you are solving unique
    • What are the known technologies?
    • Why is it not simply a matter of engineering/development?
  • Imagine you are given a budget for grocery shopping. SR&ED does not compensate for your cost of purchase
    • Justify why you had to grow your own tomato instead of buying existing ones
    • Explain why the process of growing this tomato is unknown to you
    • Capture the research effort, cost and time

40. TODOs

40.3. PHP catch fatal error then close MySQL transaction

40.6. dialpad.com, aloware.com

40.7. ClickHouse.tech

40.8. DONE MySQL sql mode

40.9. TODO Vagrant on Hyper-V

40.10. TODO CSS Nesting

40.11. TODO Inspectlet.com

40.12. TODO QUnit

40.13. TODO Learn Continuous Integration "Pantheon Build Drupal with Composer on Travis"

40.14. TODO Learn tool Zeplin

40.15. TODO Lynda Web Project Workflows with Gulp.js, Git, and Browserify

40.16. DONE Lynda Agile Project Management

40.17. TODO Lynda Firebase, Travis CI, Heroku

40.18. TODO JavaScript HTML5 Animation Framework: GSAP or Tween https://greensock.com/

40.19. TODO Lynda: MVC Frameworks for Building PHP Web Applications with Drew Falkman

40.20. TODO Book: Code Complete (2nd Edition, Microsoft Press)

40.21. TODO custom-elements-everywhere.com, williams sonoma

40.22. TODO gridbyexample.com

40.23. TODO BrowserStack and BeHat

40.24. TODO SearchKings.ca

40.25. TODO Agency DriveDigital.ca

40.26. TODO Front End Design

40.29. TODO https://grow.google/

40.33. TODO https://generalassemb.ly/ Course on Product Management

40.34. TODO https://www.roberthalf.com/ Job Hunt

40.36. TODO http://grow.io/

40.38. TODO Learn New Relic One

40.39. TODO Learn https://langserver.org

40.40. TODO Python Standard Library Essential Training

40.41. TODO Learn Rust to develop API, Restful Microservices

40.42. TODO OMCP certification

40.43. TODO Certified ScrumMaster (CSM) vs Professional Scrum Master (PSM)

40.44. TODO Book: Nonviolent Communication: A Language of Life

40.45. DONE Lynda Behance.net

40.46. DONE Lynda Illustrator CC for Web Design: Core Concepts, Aesthetics, Image Optimization, Wireframing, SVG

40.47. TODO Semantic-UI.com, Slick JS, Typed.js

41. Random

  • People
    • How many web developers? Are they responsible for all websites and enewsletters?
    • In-house Ad Ops?
    • Who will I be working with?
  • Process
    • Who will give me orders and who should I deliver my work to?
    • Any milestone I should meet? Busy time in a year
  • Product
    • websites, enewsletters, any products designed or made specifically for clients? e.g. brochure or website
    • What products will I be responsible for?
    • Do you have some goals that you want to achieve for short term and long term?
    • Key Intelligence
      • Is it a system?
    • What do you expect from me?
  • Melissa Young Sing
    • e-commerce website. Improve user experience
    • A role between Product Manager and Development Team. Documentation
    • Build an AdOps team
  • JS: object vs array
  • js implicit parameter

Author: Li Li

Created: 2024-09-05 Thu 01:12

Validate